Adding Factorization Machines #383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shauheen merged 2 commits into from
Jun 26, 2018
Merged

Adding Factorization Machines #383

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a215156
Adding Factorization Machines
sfilipi Jun 26, 2018
2b27ffe
removing utility script
sfilipi Jun 26, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions src/Microsoft.ML.CpuMath/AssemblyInfo.cs
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;

[assembly: InternalsVisibleTo("Microsoft.ML.StandardLearners, PublicKey=00240000048000009400000006020000002400005253413100040000010001004b86c4cb78549b34bab61a3b1800e23bfeb5b3ec390074041536a7e3cbd97f5f04cf0f857155a8928eaa29ebfd11cfbbad3ba70efea7bda3226c6a8d370a4cd303f714486b6ebc225985a638471e6ef571cc92a4613c00b8fa65d61ccee0cbe5f36330c9a01f4183559f1bef24cc2917c6d913e3a541333a1d05d9bed22b38cb")]

@eerhardteerhardtJun 26, 2018•
edited by sfilipi
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this necessary? #Pending

@sfilipisfilipiJun 26, 2018

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AlignedArray.Items is internal, but gets accessed in FactorizationMachines.

Will get rid of it we move off AlignedArray.

In reply to: 198213276 [](ancestors = 198213276)

94 changes: 94 additions & 0 deletions src/Microsoft.ML.StandardLearners/FactorizationMachine/FactorizationMachineInterface.cs
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using Microsoft.ML.Runtime.Internal.CpuMath;
using Microsoft.ML.Runtime.Internal.Utilities;
using System.Runtime.InteropServices;

using System.Security;

namespace Microsoft.ML.Runtime.FactorizationMachine
{
internal unsafe static class FieldAwareFactorizationMachineInterface
{
internal const string NativePath = "FactorizationMachineNative";
public const int CbAlign = 16;

private static bool Compat(AlignedArray a)

@TomFinleyTomFinleyJun 26, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @sfilipi and @wschin , could I ask, was the usage of AlignedArray here intentional, in the sense that without aligned array something bad happens? Or was it just copying over the practices in some existing code? If the latter, could we just as easily have used fixed with simple arrays, and whatnot? Relevant to discussions I've had with @eerhardt ... we want to get rid of AlignedArray.

@eerhardteerhardtJun 26, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a few small benchmark performance tests I've run, AlignedArray performs worse than using a regular array.

@wschinwschinJun 26, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FFM's core computation is done by SSE code, which requires the memory blocks to be aligned. The main computation doesn't call any member functions of AlighedArray, we just use AlighedArray as something like new operator. Thus, the performance is possibly as good as regular array.

@eerhardteerhardtJun 26, 2018•
edited by sfilipi
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my investigation, I concluded that the performance penalty being paid was where we are moving the array elements around in memory to manually align it.

machinelearning/src/Microsoft.ML.CpuMath/AlignedArray.cs

Line 67 in c023727

MoveData(ifltMin);

In my experience, doing this copying is worse performance than just using unaligned reads.

Another way to fix this issue is to use both a preamble and a postamble to get aligned. The idea is to do the operation one element at a time while you are not aligned at the beginning. Then once you hit the alignment, do the vectorization operation as usual. Then at the end, do one element at a time for the last elements that don't fill out a full section. #Pending

@sfilipisfilipiJun 26, 2018

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the note Eric. Added notes to Issue #146, to address this.

In reply to: 198231400 [](ancestors = 198231400)

{
Contracts.AssertValue(a);
Contracts.Assert(a.Size > 0);
return a.CbAlign == CbAlign;
}

private unsafe static float* Ptr(AlignedArray a, float* p)
{
Contracts.AssertValue(a);
float* q = p + a.GetBase((long)p);
Contracts.Assert(((long)q & (CbAlign - 1)) == 0);
return q;
}

[DllImport(NativePath), SuppressUnmanagedCodeSecurity]
public static extern void CalculateIntermediateVariablesNative(int fieldCount, int latentDim, int count, int* /*const*/ fieldIndices, int* /*const*/ featureIndices,
float* /*const*/ featureValues, float* /*const*/ linearWeights, float* /*const*/ latentWeights, float* latentSum, float* response);

@Ivanidzo4kaIvanidzo4kaJun 20, 2018•
edited by sfilipi
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/const/ [](start = 51, length = 9)

are this comments here for a reason? #Resolved

@sfilipisfilipiJun 20, 2018

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wschin just remove them, or need to comment on them?

In reply to: 196861706 [](ancestors = 196861706)

@wschinwschinJun 20, 2018•
edited by sfilipi
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added those comments /*const*/ following another learner's implementation. It also uses C++ and C# together. The comments means "although they are not constant pointers, you should not touch their values." #Resolved

@sfilipisfilipiJun 21, 2018

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added this comment to the code.

In reply to: 196948210 [](ancestors = 196948210)


[DllImport(NativePath), SuppressUnmanagedCodeSecurity]
public static extern void CalculateGradientAndUpdateNative(float lambdaLinear, float lambdaLatent, float learningRate, int fieldCount, int latentDim, float weight,
int count, int* /*const*/ fieldIndices, int* /*const*/ featureIndices, float* /*const*/ featureValues, float* /*const*/ latentSum, float slope,
float* linearWeights, float* latentWeights, float* linearAccumulatedSquaredGrads, float* latentAccumulatedSquaredGrads);

public static void CalculateIntermediateVariables(int fieldCount, int latentDim, int count, int[] fieldIndices, int[] featureIndices, float[] featureValues,
float[] linearWeights, AlignedArray latentWeights, AlignedArray latentSum, ref float response)
{
Contracts.AssertNonEmpty(fieldIndices);
Contracts.AssertNonEmpty(featureValues);
Contracts.AssertNonEmpty(featureIndices);
Contracts.AssertNonEmpty(linearWeights);
Contracts.Assert(Compat(latentWeights));
Contracts.Assert(Compat(latentSum));

unsafe
{
fixed (int* pf = &fieldIndices[0])
fixed (int* pi = &featureIndices[0])
fixed (float* px = &featureValues[0])
fixed (float* pw = &linearWeights[0])
fixed (float* pv = &latentWeights.Items[0])
fixed (float* pq = &latentSum.Items[0])
fixed (float* pr = &response)
CalculateIntermediateVariablesNative(fieldCount, latentDim, count, pf, pi, px, pw, Ptr(latentWeights, pv), Ptr(latentSum, pq), pr);
}
}

public static void CalculateGradientAndUpdate(float lambdaLinear, float lambdaLatent, float learningRate, int fieldCount, int latentDim,
float weight, int count, int[] fieldIndices, int[] featureIndices, float[] featureValues, AlignedArray latentSum, float slope,
float[] linearWeights, AlignedArray latentWeights, float[] linearAccumulatedSquaredGrads, AlignedArray latentAccumulatedSquaredGrads)
{
Contracts.AssertNonEmpty(fieldIndices);
Contracts.AssertNonEmpty(featureIndices);
Contracts.AssertNonEmpty(featureValues);
Contracts.Assert(Compat(latentSum));
Contracts.AssertNonEmpty(linearWeights);
Contracts.Assert(Compat(latentWeights));
Contracts.AssertNonEmpty(linearAccumulatedSquaredGrads);
Contracts.Assert(Compat(latentAccumulatedSquaredGrads));

unsafe
{
fixed (int* pf = &fieldIndices[0])
fixed (int* pi = &featureIndices[0])
fixed (float* px = &featureValues[0])
fixed (float* pq = &latentSum.Items[0])
fixed (float* pw = &linearWeights[0])
fixed (float* pv = &latentWeights.Items[0])
fixed (float* phw = &linearAccumulatedSquaredGrads[0])
fixed (float* phv = &latentAccumulatedSquaredGrads.Items[0])
CalculateGradientAndUpdateNative(lambdaLinear, lambdaLatent, learningRate, fieldCount, latentDim, weight, count, pf, pi, px,
Ptr(latentSum, pq), slope, pw, Ptr(latentWeights, pv), phw, Ptr(latentAccumulatedSquaredGrads, phv));
}

}
}
}
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Adding Factorization Machines #383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

shauheen
merged 2 commits into

from

Jun 26, 2018
 Merged


Adding Factorization Machines #383
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


2 commits


Select commit
Hold shift + click to select a range
a215156

Adding Factorization Machines


sfilipi Jun 26, 2018
2b27ffe

removing utility script


sfilipi Jun 26, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


9 changes: 9 additions & 0 deletions
src/Microsoft.ML.CpuMath/AssemblyInfo.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;

[assembly: InternalsVisibleTo("Microsoft.ML.StandardLearners, PublicKey=00240000048000009400000006020000002400005253413100040000010001004b86c4cb78549b34bab61a3b1800e23bfeb5b3ec390074041536a7e3cbd97f5f04cf0f857155a8928eaa29ebfd11cfbbad3ba70efea7bda3226c6a8d370a4cd303f714486b6ebc225985a638471e6ef571cc92a4613c00b8fa65d61ccee0cbe5f36330c9a01f4183559f1bef24cc2917c6d913e3a541333a1d05d9bed22b38cb")]
@eerhardteerhardtJun 26, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this necessary? #Pending
@sfilipisfilipiJun 26, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AlignedArray.Items is internal, but gets accessed in FactorizationMachines.
Will get rid of it we move off AlignedArray.
In reply to: 198213276 [](ancestors = 198213276)

94 changes: 94 additions & 0 deletions
src/Microsoft.ML.StandardLearners/FactorizationMachine/FactorizationMachineInterface.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using Microsoft.ML.Runtime.Internal.CpuMath;
using Microsoft.ML.Runtime.Internal.Utilities;
using System.Runtime.InteropServices;

using System.Security;

namespace Microsoft.ML.Runtime.FactorizationMachine
{
 internal unsafe static class FieldAwareFactorizationMachineInterface
 {
 internal const string NativePath = "FactorizationMachineNative";
 public const int CbAlign = 16;

 private static bool Compat(AlignedArray a)
@TomFinleyTomFinleyJun 26, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @sfilipi and @wschin , could I ask, was the usage of AlignedArray here intentional, in the sense that without aligned array something bad happens? Or was it just copying over the practices in some existing code? If the latter, could we just as easily have used fixed with simple arrays, and whatnot? Relevant to discussions I've had with @eerhardt ... we want to get rid of AlignedArray.
@eerhardteerhardtJun 26, 2018

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a few small benchmark performance tests I've run, AlignedArray performs worse than using a regular array.
@wschinwschinJun 26, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FFM's core computation is done by SSE code, which requires the memory blocks to be aligned. The main computation doesn't call any member functions of AlighedArray, we just use AlighedArray as something like new operator. Thus, the performance is possibly as good as regular array.
@eerhardteerhardtJun 26, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my investigation, I concluded that the performance penalty being paid was where we are moving the array elements around in memory to manually align it.
machinelearning/src/Microsoft.ML.CpuMath/AlignedArray.cs

Line 67
in
c023727
MoveData(ifltMin);
In my experience, doing this copying is worse performance than just using unaligned reads.
Another way to fix this issue is to use both a preamble and a postamble to get aligned. The idea is to do the operation one element at a time while you are not aligned at the beginning. Then once you hit the alignment, do the vectorization operation as usual. Then at the end, do one element at a time for the last elements that don't fill out a full section. #Pending
@sfilipisfilipiJun 26, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the note Eric. Added notes to Issue #146, to address this.
In reply to: 198231400 [](ancestors = 198231400)
 {
 Contracts.AssertValue(a);
 Contracts.Assert(a.Size > 0);
 return a.CbAlign == CbAlign;
 }

 private unsafe static float* Ptr(AlignedArray a, float* p)
 {
 Contracts.AssertValue(a);
 float* q = p + a.GetBase((long)p);
 Contracts.Assert(((long)q & (CbAlign - 1)) == 0);
 return q;
 }

 [DllImport(NativePath), SuppressUnmanagedCodeSecurity]
 public static extern void CalculateIntermediateVariablesNative(int fieldCount, int latentDim, int count, int* /*const*/ fieldIndices, int* /*const*/ featureIndices,
 float* /*const*/ featureValues, float* /*const*/ linearWeights, float* /*const*/ latentWeights, float* latentSum, float* response);
@Ivanidzo4kaIvanidzo4kaJun 20, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/const/ [](start = 51, length = 9)
are this comments here for a reason? #Resolved
@sfilipisfilipiJun 20, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wschin just remove them, or need to comment on them?
In reply to: 196861706 [](ancestors = 196861706)
@wschinwschinJun 20, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added those comments /*const*/ following another learner's implementation. It also uses C++ and C# together. The comments means "although they are not constant pointers, you should not touch their values." #Resolved
@sfilipisfilipiJun 21, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added this comment to the code.
In reply to: 196948210 [](ancestors = 196948210)

 [DllImport(NativePath), SuppressUnmanagedCodeSecurity]
 public static extern void CalculateGradientAndUpdateNative(float lambdaLinear, float lambdaLatent, float learningRate, int fieldCount, int latentDim, float weight,
 int count, int* /*const*/ fieldIndices, int* /*const*/ featureIndices, float* /*const*/ featureValues, float* /*const*/ latentSum, float slope,
 float* linearWeights, float* latentWeights, float* linearAccumulatedSquaredGrads, float* latentAccumulatedSquaredGrads);

 public static void CalculateIntermediateVariables(int fieldCount, int latentDim, int count, int[] fieldIndices, int[] featureIndices, float[] featureValues,
 float[] linearWeights, AlignedArray latentWeights, AlignedArray latentSum, ref float response)
 {
 Contracts.AssertNonEmpty(fieldIndices);
 Contracts.AssertNonEmpty(featureValues);
 Contracts.AssertNonEmpty(featureIndices);
 Contracts.AssertNonEmpty(linearWeights);
 Contracts.Assert(Compat(latentWeights));
 Contracts.Assert(Compat(latentSum));

 unsafe
 {
 fixed (int* pf = &fieldIndices[0])
 fixed (int* pi = &featureIndices[0])
 fixed (float* px = &featureValues[0])
 fixed (float* pw = &linearWeights[0])
 fixed (float* pv = &latentWeights.Items[0])
 fixed (float* pq = &latentSum.Items[0])
 fixed (float* pr = &response)
 CalculateIntermediateVariablesNative(fieldCount, latentDim, count, pf, pi, px, pw, Ptr(latentWeights, pv), Ptr(latentSum, pq), pr);
 }
 }

 public static void CalculateGradientAndUpdate(float lambdaLinear, float lambdaLatent, float learningRate, int fieldCount, int latentDim,
 float weight, int count, int[] fieldIndices, int[] featureIndices, float[] featureValues, AlignedArray latentSum, float slope,
 float[] linearWeights, AlignedArray latentWeights, float[] linearAccumulatedSquaredGrads, AlignedArray latentAccumulatedSquaredGrads)
 {
 Contracts.AssertNonEmpty(fieldIndices);
 Contracts.AssertNonEmpty(featureIndices);
 Contracts.AssertNonEmpty(featureValues);
 Contracts.Assert(Compat(latentSum));
 Contracts.AssertNonEmpty(linearWeights);
 Contracts.Assert(Compat(latentWeights));
 Contracts.AssertNonEmpty(linearAccumulatedSquaredGrads);
 Contracts.Assert(Compat(latentAccumulatedSquaredGrads));

 unsafe
 {
 fixed (int* pf = &fieldIndices[0])
 fixed (int* pi = &featureIndices[0])
 fixed (float* px = &featureValues[0])
 fixed (float* pq = &latentSum.Items[0])
 fixed (float* pw = &linearWeights[0])
 fixed (float* pv = &latentWeights.Items[0])
 fixed (float* phw = &linearAccumulatedSquaredGrads[0])
 fixed (float* phv = &latentAccumulatedSquaredGrads.Items[0])
 CalculateGradientAndUpdateNative(lambdaLinear, lambdaLatent, learningRate, fieldCount, latentDim, weight, count, pf, pi, px,
 Ptr(latentSum, pq), slope, pw, Ptr(latentWeights, pv), phw, Ptr(latentAccumulatedSquaredGrads, phv));
 }

 }
 }
}
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Adding Factorization Machines #383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

shauheen
merged 2 commits into

from

Jun 26, 2018
 Merged


Adding Factorization Machines #383
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


2 commits


Select commit
Hold shift + click to select a range
a215156

Adding Factorization Machines


sfilipi Jun 26, 2018
2b27ffe

removing utility script


sfilipi Jun 26, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


9 changes: 9 additions & 0 deletions
src/Microsoft.ML.CpuMath/AssemblyInfo.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;

[assembly: InternalsVisibleTo("Microsoft.ML.StandardLearners, PublicKey=00240000048000009400000006020000002400005253413100040000010001004b86c4cb78549b34bab61a3b1800e23bfeb5b3ec390074041536a7e3cbd97f5f04cf0f857155a8928eaa29ebfd11cfbbad3ba70efea7bda3226c6a8d370a4cd303f714486b6ebc225985a638471e6ef571cc92a4613c00b8fa65d61ccee0cbe5f36330c9a01f4183559f1bef24cc2917c6d913e3a541333a1d05d9bed22b38cb")]
@eerhardteerhardtJun 26, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this necessary? #Pending
@sfilipisfilipiJun 26, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AlignedArray.Items is internal, but gets accessed in FactorizationMachines.
Will get rid of it we move off AlignedArray.
In reply to: 198213276 [](ancestors = 198213276)

94 changes: 94 additions & 0 deletions
src/Microsoft.ML.StandardLearners/FactorizationMachine/FactorizationMachineInterface.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using Microsoft.ML.Runtime.Internal.CpuMath;
using Microsoft.ML.Runtime.Internal.Utilities;
using System.Runtime.InteropServices;

using System.Security;

namespace Microsoft.ML.Runtime.FactorizationMachine
{
 internal unsafe static class FieldAwareFactorizationMachineInterface
 {
 internal const string NativePath = "FactorizationMachineNative";
 public const int CbAlign = 16;

 private static bool Compat(AlignedArray a)
@TomFinleyTomFinleyJun 26, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @sfilipi and @wschin , could I ask, was the usage of AlignedArray here intentional, in the sense that without aligned array something bad happens? Or was it just copying over the practices in some existing code? If the latter, could we just as easily have used fixed with simple arrays, and whatnot? Relevant to discussions I've had with @eerhardt ... we want to get rid of AlignedArray.
@eerhardteerhardtJun 26, 2018

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a few small benchmark performance tests I've run, AlignedArray performs worse than using a regular array.
@wschinwschinJun 26, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FFM's core computation is done by SSE code, which requires the memory blocks to be aligned. The main computation doesn't call any member functions of AlighedArray, we just use AlighedArray as something like new operator. Thus, the performance is possibly as good as regular array.
@eerhardteerhardtJun 26, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my investigation, I concluded that the performance penalty being paid was where we are moving the array elements around in memory to manually align it.
machinelearning/src/Microsoft.ML.CpuMath/AlignedArray.cs

Line 67
in
c023727
MoveData(ifltMin);
In my experience, doing this copying is worse performance than just using unaligned reads.
Another way to fix this issue is to use both a preamble and a postamble to get aligned. The idea is to do the operation one element at a time while you are not aligned at the beginning. Then once you hit the alignment, do the vectorization operation as usual. Then at the end, do one element at a time for the last elements that don't fill out a full section. #Pending
@sfilipisfilipiJun 26, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the note Eric. Added notes to Issue #146, to address this.
In reply to: 198231400 [](ancestors = 198231400)
 {
 Contracts.AssertValue(a);
 Contracts.Assert(a.Size > 0);
 return a.CbAlign == CbAlign;
 }

 private unsafe static float* Ptr(AlignedArray a, float* p)
 {
 Contracts.AssertValue(a);
 float* q = p + a.GetBase((long)p);
 Contracts.Assert(((long)q & (CbAlign - 1)) == 0);
 return q;
 }

 [DllImport(NativePath), SuppressUnmanagedCodeSecurity]
 public static extern void CalculateIntermediateVariablesNative(int fieldCount, int latentDim, int count, int* /*const*/ fieldIndices, int* /*const*/ featureIndices,
 float* /*const*/ featureValues, float* /*const*/ linearWeights, float* /*const*/ latentWeights, float* latentSum, float* response);
@Ivanidzo4kaIvanidzo4kaJun 20, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/const/ [](start = 51, length = 9)
are this comments here for a reason? #Resolved
@sfilipisfilipiJun 20, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wschin just remove them, or need to comment on them?
In reply to: 196861706 [](ancestors = 196861706)
@wschinwschinJun 20, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added those comments /*const*/ following another learner's implementation. It also uses C++ and C# together. The comments means "although they are not constant pointers, you should not touch their values." #Resolved
@sfilipisfilipiJun 21, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added this comment to the code.
In reply to: 196948210 [](ancestors = 196948210)

 [DllImport(NativePath), SuppressUnmanagedCodeSecurity]
 public static extern void CalculateGradientAndUpdateNative(float lambdaLinear, float lambdaLatent, float learningRate, int fieldCount, int latentDim, float weight,
 int count, int* /*const*/ fieldIndices, int* /*const*/ featureIndices, float* /*const*/ featureValues, float* /*const*/ latentSum, float slope,
 float* linearWeights, float* latentWeights, float* linearAccumulatedSquaredGrads, float* latentAccumulatedSquaredGrads);

 public static void CalculateIntermediateVariables(int fieldCount, int latentDim, int count, int[] fieldIndices, int[] featureIndices, float[] featureValues,
 float[] linearWeights, AlignedArray latentWeights, AlignedArray latentSum, ref float response)
 {
 Contracts.AssertNonEmpty(fieldIndices);
 Contracts.AssertNonEmpty(featureValues);
 Contracts.AssertNonEmpty(featureIndices);
 Contracts.AssertNonEmpty(linearWeights);
 Contracts.Assert(Compat(latentWeights));
 Contracts.Assert(Compat(latentSum));

 unsafe
 {
 fixed (int* pf = &fieldIndices[0])
 fixed (int* pi = &featureIndices[0])
 fixed (float* px = &featureValues[0])
 fixed (float* pw = &linearWeights[0])
 fixed (float* pv = &latentWeights.Items[0])
 fixed (float* pq = &latentSum.Items[0])
 fixed (float* pr = &response)
 CalculateIntermediateVariablesNative(fieldCount, latentDim, count, pf, pi, px, pw, Ptr(latentWeights, pv), Ptr(latentSum, pq), pr);
 }
 }

 public static void CalculateGradientAndUpdate(float lambdaLinear, float lambdaLatent, float learningRate, int fieldCount, int latentDim,
 float weight, int count, int[] fieldIndices, int[] featureIndices, float[] featureValues, AlignedArray latentSum, float slope,
 float[] linearWeights, AlignedArray latentWeights, float[] linearAccumulatedSquaredGrads, AlignedArray latentAccumulatedSquaredGrads)
 {
 Contracts.AssertNonEmpty(fieldIndices);
 Contracts.AssertNonEmpty(featureIndices);
 Contracts.AssertNonEmpty(featureValues);
 Contracts.Assert(Compat(latentSum));
 Contracts.AssertNonEmpty(linearWeights);
 Contracts.Assert(Compat(latentWeights));
 Contracts.AssertNonEmpty(linearAccumulatedSquaredGrads);
 Contracts.Assert(Compat(latentAccumulatedSquaredGrads));

 unsafe
 {
 fixed (int* pf = &fieldIndices[0])
 fixed (int* pi = &featureIndices[0])
 fixed (float* px = &featureValues[0])
 fixed (float* pq = &latentSum.Items[0])
 fixed (float* pw = &linearWeights[0])
 fixed (float* pv = &latentWeights.Items[0])
 fixed (float* phw = &linearAccumulatedSquaredGrads[0])
 fixed (float* phv = &latentAccumulatedSquaredGrads.Items[0])
 CalculateGradientAndUpdateNative(lambdaLinear, lambdaLatent, learningRate, fieldCount, latentDim, weight, count, pf, pi, px,
 Ptr(latentSum, pq), slope, pw, Ptr(latentWeights, pv), phw, Ptr(latentAccumulatedSquaredGrads, phv));
 }

 }
 }
}
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Adding Factorization Machines #383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

shauheen
merged 2 commits into

from

Jun 26, 2018
 Merged


Adding Factorization Machines #383
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


2 commits


Select commit
Hold shift + click to select a range
a215156

Adding Factorization Machines


sfilipi Jun 26, 2018
2b27ffe

removing utility script


sfilipi Jun 26, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


9 changes: 9 additions & 0 deletions
src/Microsoft.ML.CpuMath/AssemblyInfo.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;

[assembly: InternalsVisibleTo("Microsoft.ML.StandardLearners, PublicKey=00240000048000009400000006020000002400005253413100040000010001004b86c4cb78549b34bab61a3b1800e23bfeb5b3ec390074041536a7e3cbd97f5f04cf0f857155a8928eaa29ebfd11cfbbad3ba70efea7bda3226c6a8d370a4cd303f714486b6ebc225985a638471e6ef571cc92a4613c00b8fa65d61ccee0cbe5f36330c9a01f4183559f1bef24cc2917c6d913e3a541333a1d05d9bed22b38cb")]
@eerhardteerhardtJun 26, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this necessary? #Pending
@sfilipisfilipiJun 26, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AlignedArray.Items is internal, but gets accessed in FactorizationMachines.
Will get rid of it we move off AlignedArray.
In reply to: 198213276 [](ancestors = 198213276)

94 changes: 94 additions & 0 deletions
src/Microsoft.ML.StandardLearners/FactorizationMachine/FactorizationMachineInterface.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using Microsoft.ML.Runtime.Internal.CpuMath;
using Microsoft.ML.Runtime.Internal.Utilities;
using System.Runtime.InteropServices;

using System.Security;

namespace Microsoft.ML.Runtime.FactorizationMachine
{
 internal unsafe static class FieldAwareFactorizationMachineInterface
 {
 internal const string NativePath = "FactorizationMachineNative";
 public const int CbAlign = 16;

 private static bool Compat(AlignedArray a)
@TomFinleyTomFinleyJun 26, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @sfilipi and @wschin , could I ask, was the usage of AlignedArray here intentional, in the sense that without aligned array something bad happens? Or was it just copying over the practices in some existing code? If the latter, could we just as easily have used fixed with simple arrays, and whatnot? Relevant to discussions I've had with @eerhardt ... we want to get rid of AlignedArray.
@eerhardteerhardtJun 26, 2018

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a few small benchmark performance tests I've run, AlignedArray performs worse than using a regular array.
@wschinwschinJun 26, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FFM's core computation is done by SSE code, which requires the memory blocks to be aligned. The main computation doesn't call any member functions of AlighedArray, we just use AlighedArray as something like new operator. Thus, the performance is possibly as good as regular array.
@eerhardteerhardtJun 26, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my investigation, I concluded that the performance penalty being paid was where we are moving the array elements around in memory to manually align it.
machinelearning/src/Microsoft.ML.CpuMath/AlignedArray.cs

Line 67
in
c023727
MoveData(ifltMin);
In my experience, doing this copying is worse performance than just using unaligned reads.
Another way to fix this issue is to use both a preamble and a postamble to get aligned. The idea is to do the operation one element at a time while you are not aligned at the beginning. Then once you hit the alignment, do the vectorization operation as usual. Then at the end, do one element at a time for the last elements that don't fill out a full section. #Pending
@sfilipisfilipiJun 26, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the note Eric. Added notes to Issue #146, to address this.
In reply to: 198231400 [](ancestors = 198231400)
 {
 Contracts.AssertValue(a);
 Contracts.Assert(a.Size > 0);
 return a.CbAlign == CbAlign;
 }

 private unsafe static float* Ptr(AlignedArray a, float* p)
 {
 Contracts.AssertValue(a);
 float* q = p + a.GetBase((long)p);
 Contracts.Assert(((long)q & (CbAlign - 1)) == 0);
 return q;
 }

 [DllImport(NativePath), SuppressUnmanagedCodeSecurity]
 public static extern void CalculateIntermediateVariablesNative(int fieldCount, int latentDim, int count, int* /*const*/ fieldIndices, int* /*const*/ featureIndices,
 float* /*const*/ featureValues, float* /*const*/ linearWeights, float* /*const*/ latentWeights, float* latentSum, float* response);
@Ivanidzo4kaIvanidzo4kaJun 20, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/const/ [](start = 51, length = 9)
are this comments here for a reason? #Resolved
@sfilipisfilipiJun 20, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wschin just remove them, or need to comment on them?
In reply to: 196861706 [](ancestors = 196861706)
@wschinwschinJun 20, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added those comments /*const*/ following another learner's implementation. It also uses C++ and C# together. The comments means "although they are not constant pointers, you should not touch their values." #Resolved
@sfilipisfilipiJun 21, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added this comment to the code.
In reply to: 196948210 [](ancestors = 196948210)

 [DllImport(NativePath), SuppressUnmanagedCodeSecurity]
 public static extern void CalculateGradientAndUpdateNative(float lambdaLinear, float lambdaLatent, float learningRate, int fieldCount, int latentDim, float weight,
 int count, int* /*const*/ fieldIndices, int* /*const*/ featureIndices, float* /*const*/ featureValues, float* /*const*/ latentSum, float slope,
 float* linearWeights, float* latentWeights, float* linearAccumulatedSquaredGrads, float* latentAccumulatedSquaredGrads);

 public static void CalculateIntermediateVariables(int fieldCount, int latentDim, int count, int[] fieldIndices, int[] featureIndices, float[] featureValues,
 float[] linearWeights, AlignedArray latentWeights, AlignedArray latentSum, ref float response)
 {
 Contracts.AssertNonEmpty(fieldIndices);
 Contracts.AssertNonEmpty(featureValues);
 Contracts.AssertNonEmpty(featureIndices);
 Contracts.AssertNonEmpty(linearWeights);
 Contracts.Assert(Compat(latentWeights));
 Contracts.Assert(Compat(latentSum));

 unsafe
 {
 fixed (int* pf = &fieldIndices[0])
 fixed (int* pi = &featureIndices[0])
 fixed (float* px = &featureValues[0])
 fixed (float* pw = &linearWeights[0])
 fixed (float* pv = &latentWeights.Items[0])
 fixed (float* pq = &latentSum.Items[0])
 fixed (float* pr = &response)
 CalculateIntermediateVariablesNative(fieldCount, latentDim, count, pf, pi, px, pw, Ptr(latentWeights, pv), Ptr(latentSum, pq), pr);
 }
 }

 public static void CalculateGradientAndUpdate(float lambdaLinear, float lambdaLatent, float learningRate, int fieldCount, int latentDim,
 float weight, int count, int[] fieldIndices, int[] featureIndices, float[] featureValues, AlignedArray latentSum, float slope,
 float[] linearWeights, AlignedArray latentWeights, float[] linearAccumulatedSquaredGrads, AlignedArray latentAccumulatedSquaredGrads)
 {
 Contracts.AssertNonEmpty(fieldIndices);
 Contracts.AssertNonEmpty(featureIndices);
 Contracts.AssertNonEmpty(featureValues);
 Contracts.Assert(Compat(latentSum));
 Contracts.AssertNonEmpty(linearWeights);
 Contracts.Assert(Compat(latentWeights));
 Contracts.AssertNonEmpty(linearAccumulatedSquaredGrads);
 Contracts.Assert(Compat(latentAccumulatedSquaredGrads));

 unsafe
 {
 fixed (int* pf = &fieldIndices[0])
 fixed (int* pi = &featureIndices[0])
 fixed (float* px = &featureValues[0])
 fixed (float* pq = &latentSum.Items[0])
 fixed (float* pw = &linearWeights[0])
 fixed (float* pv = &latentWeights.Items[0])
 fixed (float* phw = &linearAccumulatedSquaredGrads[0])
 fixed (float* phv = &latentAccumulatedSquaredGrads.Items[0])
 CalculateGradientAndUpdateNative(lambdaLinear, lambdaLatent, learningRate, fieldCount, latentDim, weight, count, pf, pi, px,
 Ptr(latentSum, pq), slope, pw, Ptr(latentWeights, pv), phw, Ptr(latentAccumulatedSquaredGrads, phv));
 }

 }
 }
}
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Adding Factorization Machines #383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

shauheen
merged 2 commits into

from

Jun 26, 2018
 Merged


Adding Factorization Machines #383
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


2 commits


Select commit
Hold shift + click to select a range
a215156

Adding Factorization Machines


sfilipi Jun 26, 2018
2b27ffe

removing utility script


sfilipi Jun 26, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


9 changes: 9 additions & 0 deletions
src/Microsoft.ML.CpuMath/AssemblyInfo.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;

[assembly: InternalsVisibleTo("Microsoft.ML.StandardLearners, PublicKey=00240000048000009400000006020000002400005253413100040000010001004b86c4cb78549b34bab61a3b1800e23bfeb5b3ec390074041536a7e3cbd97f5f04cf0f857155a8928eaa29ebfd11cfbbad3ba70efea7bda3226c6a8d370a4cd303f714486b6ebc225985a638471e6ef571cc92a4613c00b8fa65d61ccee0cbe5f36330c9a01f4183559f1bef24cc2917c6d913e3a541333a1d05d9bed22b38cb")]
@eerhardteerhardtJun 26, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this necessary? #Pending
@sfilipisfilipiJun 26, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AlignedArray.Items is internal, but gets accessed in FactorizationMachines.
Will get rid of it we move off AlignedArray.
In reply to: 198213276 [](ancestors = 198213276)

94 changes: 94 additions & 0 deletions
src/Microsoft.ML.StandardLearners/FactorizationMachine/FactorizationMachineInterface.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using Microsoft.ML.Runtime.Internal.CpuMath;
using Microsoft.ML.Runtime.Internal.Utilities;
using System.Runtime.InteropServices;

using System.Security;

namespace Microsoft.ML.Runtime.FactorizationMachine
{
 internal unsafe static class FieldAwareFactorizationMachineInterface
 {
 internal const string NativePath = "FactorizationMachineNative";
 public const int CbAlign = 16;

 private static bool Compat(AlignedArray a)
@TomFinleyTomFinleyJun 26, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @sfilipi and @wschin , could I ask, was the usage of AlignedArray here intentional, in the sense that without aligned array something bad happens? Or was it just copying over the practices in some existing code? If the latter, could we just as easily have used fixed with simple arrays, and whatnot? Relevant to discussions I've had with @eerhardt ... we want to get rid of AlignedArray.
@eerhardteerhardtJun 26, 2018

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a few small benchmark performance tests I've run, AlignedArray performs worse than using a regular array.
@wschinwschinJun 26, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FFM's core computation is done by SSE code, which requires the memory blocks to be aligned. The main computation doesn't call any member functions of AlighedArray, we just use AlighedArray as something like new operator. Thus, the performance is possibly as good as regular array.
@eerhardteerhardtJun 26, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my investigation, I concluded that the performance penalty being paid was where we are moving the array elements around in memory to manually align it.
machinelearning/src/Microsoft.ML.CpuMath/AlignedArray.cs

Line 67
in
c023727
MoveData(ifltMin);
In my experience, doing this copying is worse performance than just using unaligned reads.
Another way to fix this issue is to use both a preamble and a postamble to get aligned. The idea is to do the operation one element at a time while you are not aligned at the beginning. Then once you hit the alignment, do the vectorization operation as usual. Then at the end, do one element at a time for the last elements that don't fill out a full section. #Pending
@sfilipisfilipiJun 26, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the note Eric. Added notes to Issue #146, to address this.
In reply to: 198231400 [](ancestors = 198231400)
 {
 Contracts.AssertValue(a);
 Contracts.Assert(a.Size > 0);
 return a.CbAlign == CbAlign;
 }

 private unsafe static float* Ptr(AlignedArray a, float* p)
 {
 Contracts.AssertValue(a);
 float* q = p + a.GetBase((long)p);
 Contracts.Assert(((long)q & (CbAlign - 1)) == 0);
 return q;
 }

 [DllImport(NativePath), SuppressUnmanagedCodeSecurity]
 public static extern void CalculateIntermediateVariablesNative(int fieldCount, int latentDim, int count, int* /*const*/ fieldIndices, int* /*const*/ featureIndices,
 float* /*const*/ featureValues, float* /*const*/ linearWeights, float* /*const*/ latentWeights, float* latentSum, float* response);
@Ivanidzo4kaIvanidzo4kaJun 20, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/const/ [](start = 51, length = 9)
are this comments here for a reason? #Resolved
@sfilipisfilipiJun 20, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wschin just remove them, or need to comment on them?
In reply to: 196861706 [](ancestors = 196861706)
@wschinwschinJun 20, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added those comments /*const*/ following another learner's implementation. It also uses C++ and C# together. The comments means "although they are not constant pointers, you should not touch their values." #Resolved
@sfilipisfilipiJun 21, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added this comment to the code.
In reply to: 196948210 [](ancestors = 196948210)

 [DllImport(NativePath), SuppressUnmanagedCodeSecurity]
 public static extern void CalculateGradientAndUpdateNative(float lambdaLinear, float lambdaLatent, float learningRate, int fieldCount, int latentDim, float weight,
 int count, int* /*const*/ fieldIndices, int* /*const*/ featureIndices, float* /*const*/ featureValues, float* /*const*/ latentSum, float slope,
 float* linearWeights, float* latentWeights, float* linearAccumulatedSquaredGrads, float* latentAccumulatedSquaredGrads);

 public static void CalculateIntermediateVariables(int fieldCount, int latentDim, int count, int[] fieldIndices, int[] featureIndices, float[] featureValues,
 float[] linearWeights, AlignedArray latentWeights, AlignedArray latentSum, ref float response)
 {
 Contracts.AssertNonEmpty(fieldIndices);
 Contracts.AssertNonEmpty(featureValues);
 Contracts.AssertNonEmpty(featureIndices);
 Contracts.AssertNonEmpty(linearWeights);
 Contracts.Assert(Compat(latentWeights));
 Contracts.Assert(Compat(latentSum));

 unsafe
 {
 fixed (int* pf = &fieldIndices[0])
 fixed (int* pi = &featureIndices[0])
 fixed (float* px = &featureValues[0])
 fixed (float* pw = &linearWeights[0])
 fixed (float* pv = &latentWeights.Items[0])
 fixed (float* pq = &latentSum.Items[0])
 fixed (float* pr = &response)
 CalculateIntermediateVariablesNative(fieldCount, latentDim, count, pf, pi, px, pw, Ptr(latentWeights, pv), Ptr(latentSum, pq), pr);
 }
 }

 public static void CalculateGradientAndUpdate(float lambdaLinear, float lambdaLatent, float learningRate, int fieldCount, int latentDim,
 float weight, int count, int[] fieldIndices, int[] featureIndices, float[] featureValues, AlignedArray latentSum, float slope,
 float[] linearWeights, AlignedArray latentWeights, float[] linearAccumulatedSquaredGrads, AlignedArray latentAccumulatedSquaredGrads)
 {
 Contracts.AssertNonEmpty(fieldIndices);
 Contracts.AssertNonEmpty(featureIndices);
 Contracts.AssertNonEmpty(featureValues);
 Contracts.Assert(Compat(latentSum));
 Contracts.AssertNonEmpty(linearWeights);
 Contracts.Assert(Compat(latentWeights));
 Contracts.AssertNonEmpty(linearAccumulatedSquaredGrads);
 Contracts.Assert(Compat(latentAccumulatedSquaredGrads));

 unsafe
 {
 fixed (int* pf = &fieldIndices[0])
 fixed (int* pi = &featureIndices[0])
 fixed (float* px = &featureValues[0])
 fixed (float* pq = &latentSum.Items[0])
 fixed (float* pw = &linearWeights[0])
 fixed (float* pv = &latentWeights.Items[0])
 fixed (float* phw = &linearAccumulatedSquaredGrads[0])
 fixed (float* phv = &latentAccumulatedSquaredGrads.Items[0])
 CalculateGradientAndUpdateNative(lambdaLinear, lambdaLatent, learningRate, fieldCount, latentDim, weight, count, pf, pi, px,
 Ptr(latentSum, pq), slope, pw, Ptr(latentWeights, pv), phw, Ptr(latentAccumulatedSquaredGrads, phv));
 }

 }
 }
}
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Adding Factorization Machines #383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

shauheen
merged 2 commits into

from

Jun 26, 2018
 Merged


Adding Factorization Machines #383
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


2 commits


Select commit
Hold shift + click to select a range
a215156

Adding Factorization Machines


sfilipi Jun 26, 2018
2b27ffe

removing utility script


sfilipi Jun 26, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


9 changes: 9 additions & 0 deletions
src/Microsoft.ML.CpuMath/AssemblyInfo.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;

[assembly: InternalsVisibleTo("Microsoft.ML.StandardLearners, PublicKey=00240000048000009400000006020000002400005253413100040000010001004b86c4cb78549b34bab61a3b1800e23bfeb5b3ec390074041536a7e3cbd97f5f04cf0f857155a8928eaa29ebfd11cfbbad3ba70efea7bda3226c6a8d370a4cd303f714486b6ebc225985a638471e6ef571cc92a4613c00b8fa65d61ccee0cbe5f36330c9a01f4183559f1bef24cc2917c6d913e3a541333a1d05d9bed22b38cb")]
@eerhardteerhardtJun 26, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this necessary? #Pending
@sfilipisfilipiJun 26, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AlignedArray.Items is internal, but gets accessed in FactorizationMachines.
Will get rid of it we move off AlignedArray.
In reply to: 198213276 [](ancestors = 198213276)

94 changes: 94 additions & 0 deletions
src/Microsoft.ML.StandardLearners/FactorizationMachine/FactorizationMachineInterface.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using Microsoft.ML.Runtime.Internal.CpuMath;
using Microsoft.ML.Runtime.Internal.Utilities;
using System.Runtime.InteropServices;

using System.Security;

namespace Microsoft.ML.Runtime.FactorizationMachine
{
 internal unsafe static class FieldAwareFactorizationMachineInterface
 {
 internal const string NativePath = "FactorizationMachineNative";
 public const int CbAlign = 16;

 private static bool Compat(AlignedArray a)
@TomFinleyTomFinleyJun 26, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @sfilipi and @wschin , could I ask, was the usage of AlignedArray here intentional, in the sense that without aligned array something bad happens? Or was it just copying over the practices in some existing code? If the latter, could we just as easily have used fixed with simple arrays, and whatnot? Relevant to discussions I've had with @eerhardt ... we want to get rid of AlignedArray.
@eerhardteerhardtJun 26, 2018

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a few small benchmark performance tests I've run, AlignedArray performs worse than using a regular array.
@wschinwschinJun 26, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FFM's core computation is done by SSE code, which requires the memory blocks to be aligned. The main computation doesn't call any member functions of AlighedArray, we just use AlighedArray as something like new operator. Thus, the performance is possibly as good as regular array.
@eerhardteerhardtJun 26, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my investigation, I concluded that the performance penalty being paid was where we are moving the array elements around in memory to manually align it.
machinelearning/src/Microsoft.ML.CpuMath/AlignedArray.cs

Line 67
in
c023727
MoveData(ifltMin);
In my experience, doing this copying is worse performance than just using unaligned reads.
Another way to fix this issue is to use both a preamble and a postamble to get aligned. The idea is to do the operation one element at a time while you are not aligned at the beginning. Then once you hit the alignment, do the vectorization operation as usual. Then at the end, do one element at a time for the last elements that don't fill out a full section. #Pending
@sfilipisfilipiJun 26, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the note Eric. Added notes to Issue #146, to address this.
In reply to: 198231400 [](ancestors = 198231400)
 {
 Contracts.AssertValue(a);
 Contracts.Assert(a.Size > 0);
 return a.CbAlign == CbAlign;
 }

 private unsafe static float* Ptr(AlignedArray a, float* p)
 {
 Contracts.AssertValue(a);
 float* q = p + a.GetBase((long)p);
 Contracts.Assert(((long)q & (CbAlign - 1)) == 0);
 return q;
 }

 [DllImport(NativePath), SuppressUnmanagedCodeSecurity]
 public static extern void CalculateIntermediateVariablesNative(int fieldCount, int latentDim, int count, int* /*const*/ fieldIndices, int* /*const*/ featureIndices,
 float* /*const*/ featureValues, float* /*const*/ linearWeights, float* /*const*/ latentWeights, float* latentSum, float* response);
@Ivanidzo4kaIvanidzo4kaJun 20, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/const/ [](start = 51, length = 9)
are this comments here for a reason? #Resolved
@sfilipisfilipiJun 20, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wschin just remove them, or need to comment on them?
In reply to: 196861706 [](ancestors = 196861706)
@wschinwschinJun 20, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added those comments /*const*/ following another learner's implementation. It also uses C++ and C# together. The comments means "although they are not constant pointers, you should not touch their values." #Resolved
@sfilipisfilipiJun 21, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added this comment to the code.
In reply to: 196948210 [](ancestors = 196948210)

 [DllImport(NativePath), SuppressUnmanagedCodeSecurity]
 public static extern void CalculateGradientAndUpdateNative(float lambdaLinear, float lambdaLatent, float learningRate, int fieldCount, int latentDim, float weight,
 int count, int* /*const*/ fieldIndices, int* /*const*/ featureIndices, float* /*const*/ featureValues, float* /*const*/ latentSum, float slope,
 float* linearWeights, float* latentWeights, float* linearAccumulatedSquaredGrads, float* latentAccumulatedSquaredGrads);

 public static void CalculateIntermediateVariables(int fieldCount, int latentDim, int count, int[] fieldIndices, int[] featureIndices, float[] featureValues,
 float[] linearWeights, AlignedArray latentWeights, AlignedArray latentSum, ref float response)
 {
 Contracts.AssertNonEmpty(fieldIndices);
 Contracts.AssertNonEmpty(featureValues);
 Contracts.AssertNonEmpty(featureIndices);
 Contracts.AssertNonEmpty(linearWeights);
 Contracts.Assert(Compat(latentWeights));
 Contracts.Assert(Compat(latentSum));

 unsafe
 {
 fixed (int* pf = &fieldIndices[0])
 fixed (int* pi = &featureIndices[0])
 fixed (float* px = &featureValues[0])
 fixed (float* pw = &linearWeights[0])
 fixed (float* pv = &latentWeights.Items[0])
 fixed (float* pq = &latentSum.Items[0])
 fixed (float* pr = &response)
 CalculateIntermediateVariablesNative(fieldCount, latentDim, count, pf, pi, px, pw, Ptr(latentWeights, pv), Ptr(latentSum, pq), pr);
 }
 }

 public static void CalculateGradientAndUpdate(float lambdaLinear, float lambdaLatent, float learningRate, int fieldCount, int latentDim,
 float weight, int count, int[] fieldIndices, int[] featureIndices, float[] featureValues, AlignedArray latentSum, float slope,
 float[] linearWeights, AlignedArray latentWeights, float[] linearAccumulatedSquaredGrads, AlignedArray latentAccumulatedSquaredGrads)
 {
 Contracts.AssertNonEmpty(fieldIndices);
 Contracts.AssertNonEmpty(featureIndices);
 Contracts.AssertNonEmpty(featureValues);
 Contracts.Assert(Compat(latentSum));
 Contracts.AssertNonEmpty(linearWeights);
 Contracts.Assert(Compat(latentWeights));
 Contracts.AssertNonEmpty(linearAccumulatedSquaredGrads);
 Contracts.Assert(Compat(latentAccumulatedSquaredGrads));

 unsafe
 {
 fixed (int* pf = &fieldIndices[0])
 fixed (int* pi = &featureIndices[0])
 fixed (float* px = &featureValues[0])
 fixed (float* pq = &latentSum.Items[0])
 fixed (float* pw = &linearWeights[0])
 fixed (float* pv = &latentWeights.Items[0])
 fixed (float* phw = &linearAccumulatedSquaredGrads[0])
 fixed (float* phv = &latentAccumulatedSquaredGrads.Items[0])
 CalculateGradientAndUpdateNative(lambdaLinear, lambdaLatent, learningRate, fieldCount, latentDim, weight, count, pf, pi, px,
 Ptr(latentSum, pq), slope, pw, Ptr(latentWeights, pv), phw, Ptr(latentAccumulatedSquaredGrads, phv));
 }

 }
 }
}
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Adding Factorization Machines #383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

shauheen
merged 2 commits into

from

Jun 26, 2018
 Merged


Adding Factorization Machines #383
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


2 commits


Select commit
Hold shift + click to select a range
a215156

Adding Factorization Machines


sfilipi Jun 26, 2018
2b27ffe

removing utility script


sfilipi Jun 26, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


9 changes: 9 additions & 0 deletions
src/Microsoft.ML.CpuMath/AssemblyInfo.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;

[assembly: InternalsVisibleTo("Microsoft.ML.StandardLearners, PublicKey=00240000048000009400000006020000002400005253413100040000010001004b86c4cb78549b34bab61a3b1800e23bfeb5b3ec390074041536a7e3cbd97f5f04cf0f857155a8928eaa29ebfd11cfbbad3ba70efea7bda3226c6a8d370a4cd303f714486b6ebc225985a638471e6ef571cc92a4613c00b8fa65d61ccee0cbe5f36330c9a01f4183559f1bef24cc2917c6d913e3a541333a1d05d9bed22b38cb")]
@eerhardteerhardtJun 26, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this necessary? #Pending
@sfilipisfilipiJun 26, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AlignedArray.Items is internal, but gets accessed in FactorizationMachines.
Will get rid of it we move off AlignedArray.
In reply to: 198213276 [](ancestors = 198213276)

94 changes: 94 additions & 0 deletions
src/Microsoft.ML.StandardLearners/FactorizationMachine/FactorizationMachineInterface.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using Microsoft.ML.Runtime.Internal.CpuMath;
using Microsoft.ML.Runtime.Internal.Utilities;
using System.Runtime.InteropServices;

using System.Security;

namespace Microsoft.ML.Runtime.FactorizationMachine
{
 internal unsafe static class FieldAwareFactorizationMachineInterface
 {
 internal const string NativePath = "FactorizationMachineNative";
 public const int CbAlign = 16;

 private static bool Compat(AlignedArray a)
@TomFinleyTomFinleyJun 26, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @sfilipi and @wschin , could I ask, was the usage of AlignedArray here intentional, in the sense that without aligned array something bad happens? Or was it just copying over the practices in some existing code? If the latter, could we just as easily have used fixed with simple arrays, and whatnot? Relevant to discussions I've had with @eerhardt ... we want to get rid of AlignedArray.
@eerhardteerhardtJun 26, 2018

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a few small benchmark performance tests I've run, AlignedArray performs worse than using a regular array.
@wschinwschinJun 26, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FFM's core computation is done by SSE code, which requires the memory blocks to be aligned. The main computation doesn't call any member functions of AlighedArray, we just use AlighedArray as something like new operator. Thus, the performance is possibly as good as regular array.
@eerhardteerhardtJun 26, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my investigation, I concluded that the performance penalty being paid was where we are moving the array elements around in memory to manually align it.
machinelearning/src/Microsoft.ML.CpuMath/AlignedArray.cs

Line 67
in
c023727
MoveData(ifltMin);
In my experience, doing this copying is worse performance than just using unaligned reads.
Another way to fix this issue is to use both a preamble and a postamble to get aligned. The idea is to do the operation one element at a time while you are not aligned at the beginning. Then once you hit the alignment, do the vectorization operation as usual. Then at the end, do one element at a time for the last elements that don't fill out a full section. #Pending
@sfilipisfilipiJun 26, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the note Eric. Added notes to Issue #146, to address this.
In reply to: 198231400 [](ancestors = 198231400)
 {
 Contracts.AssertValue(a);
 Contracts.Assert(a.Size > 0);
 return a.CbAlign == CbAlign;
 }

 private unsafe static float* Ptr(AlignedArray a, float* p)
 {
 Contracts.AssertValue(a);
 float* q = p + a.GetBase((long)p);
 Contracts.Assert(((long)q & (CbAlign - 1)) == 0);
 return q;
 }

 [DllImport(NativePath), SuppressUnmanagedCodeSecurity]
 public static extern void CalculateIntermediateVariablesNative(int fieldCount, int latentDim, int count, int* /*const*/ fieldIndices, int* /*const*/ featureIndices,
 float* /*const*/ featureValues, float* /*const*/ linearWeights, float* /*const*/ latentWeights, float* latentSum, float* response);
@Ivanidzo4kaIvanidzo4kaJun 20, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/const/ [](start = 51, length = 9)
are this comments here for a reason? #Resolved
@sfilipisfilipiJun 20, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wschin just remove them, or need to comment on them?
In reply to: 196861706 [](ancestors = 196861706)
@wschinwschinJun 20, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added those comments /*const*/ following another learner's implementation. It also uses C++ and C# together. The comments means "although they are not constant pointers, you should not touch their values." #Resolved
@sfilipisfilipiJun 21, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added this comment to the code.
In reply to: 196948210 [](ancestors = 196948210)

 [DllImport(NativePath), SuppressUnmanagedCodeSecurity]
 public static extern void CalculateGradientAndUpdateNative(float lambdaLinear, float lambdaLatent, float learningRate, int fieldCount, int latentDim, float weight,
 int count, int* /*const*/ fieldIndices, int* /*const*/ featureIndices, float* /*const*/ featureValues, float* /*const*/ latentSum, float slope,
 float* linearWeights, float* latentWeights, float* linearAccumulatedSquaredGrads, float* latentAccumulatedSquaredGrads);

 public static void CalculateIntermediateVariables(int fieldCount, int latentDim, int count, int[] fieldIndices, int[] featureIndices, float[] featureValues,
 float[] linearWeights, AlignedArray latentWeights, AlignedArray latentSum, ref float response)
 {
 Contracts.AssertNonEmpty(fieldIndices);
 Contracts.AssertNonEmpty(featureValues);
 Contracts.AssertNonEmpty(featureIndices);
 Contracts.AssertNonEmpty(linearWeights);
 Contracts.Assert(Compat(latentWeights));
 Contracts.Assert(Compat(latentSum));

 unsafe
 {
 fixed (int* pf = &fieldIndices[0])
 fixed (int* pi = &featureIndices[0])
 fixed (float* px = &featureValues[0])
 fixed (float* pw = &linearWeights[0])
 fixed (float* pv = &latentWeights.Items[0])
 fixed (float* pq = &latentSum.Items[0])
 fixed (float* pr = &response)
 CalculateIntermediateVariablesNative(fieldCount, latentDim, count, pf, pi, px, pw, Ptr(latentWeights, pv), Ptr(latentSum, pq), pr);
 }
 }

 public static void CalculateGradientAndUpdate(float lambdaLinear, float lambdaLatent, float learningRate, int fieldCount, int latentDim,
 float weight, int count, int[] fieldIndices, int[] featureIndices, float[] featureValues, AlignedArray latentSum, float slope,
 float[] linearWeights, AlignedArray latentWeights, float[] linearAccumulatedSquaredGrads, AlignedArray latentAccumulatedSquaredGrads)
 {
 Contracts.AssertNonEmpty(fieldIndices);
 Contracts.AssertNonEmpty(featureIndices);
 Contracts.AssertNonEmpty(featureValues);
 Contracts.Assert(Compat(latentSum));
 Contracts.AssertNonEmpty(linearWeights);
 Contracts.Assert(Compat(latentWeights));
 Contracts.AssertNonEmpty(linearAccumulatedSquaredGrads);
 Contracts.Assert(Compat(latentAccumulatedSquaredGrads));

 unsafe
 {
 fixed (int* pf = &fieldIndices[0])
 fixed (int* pi = &featureIndices[0])
 fixed (float* px = &featureValues[0])
 fixed (float* pq = &latentSum.Items[0])
 fixed (float* pw = &linearWeights[0])
 fixed (float* pv = &latentWeights.Items[0])
 fixed (float* phw = &linearAccumulatedSquaredGrads[0])
 fixed (float* phv = &latentAccumulatedSquaredGrads.Items[0])
 CalculateGradientAndUpdateNative(lambdaLinear, lambdaLatent, learningRate, fieldCount, latentDim, weight, count, pf, pi, px,
 Ptr(latentSum, pq), slope, pw, Ptr(latentWeights, pv), phw, Ptr(latentAccumulatedSquaredGrads, phv));
 }

 }
 }
}
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Adding Factorization Machines #383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

shauheen
merged 2 commits into

from

Jun 26, 2018
 Merged


Adding Factorization Machines #383
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


2 commits


Select commit
Hold shift + click to select a range
a215156

Adding Factorization Machines


sfilipi Jun 26, 2018
2b27ffe

removing utility script


sfilipi Jun 26, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


9 changes: 9 additions & 0 deletions
src/Microsoft.ML.CpuMath/AssemblyInfo.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;

[assembly: InternalsVisibleTo("Microsoft.ML.StandardLearners, PublicKey=00240000048000009400000006020000002400005253413100040000010001004b86c4cb78549b34bab61a3b1800e23bfeb5b3ec390074041536a7e3cbd97f5f04cf0f857155a8928eaa29ebfd11cfbbad3ba70efea7bda3226c6a8d370a4cd303f714486b6ebc225985a638471e6ef571cc92a4613c00b8fa65d61ccee0cbe5f36330c9a01f4183559f1bef24cc2917c6d913e3a541333a1d05d9bed22b38cb")]
@eerhardteerhardtJun 26, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this necessary? #Pending
@sfilipisfilipiJun 26, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AlignedArray.Items is internal, but gets accessed in FactorizationMachines.
Will get rid of it we move off AlignedArray.
In reply to: 198213276 [](ancestors = 198213276)

94 changes: 94 additions & 0 deletions
src/Microsoft.ML.StandardLearners/FactorizationMachine/FactorizationMachineInterface.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using Microsoft.ML.Runtime.Internal.CpuMath;
using Microsoft.ML.Runtime.Internal.Utilities;
using System.Runtime.InteropServices;

using System.Security;

namespace Microsoft.ML.Runtime.FactorizationMachine
{
 internal unsafe static class FieldAwareFactorizationMachineInterface
 {
 internal const string NativePath = "FactorizationMachineNative";
 public const int CbAlign = 16;

 private static bool Compat(AlignedArray a)
@TomFinleyTomFinleyJun 26, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @sfilipi and @wschin , could I ask, was the usage of AlignedArray here intentional, in the sense that without aligned array something bad happens? Or was it just copying over the practices in some existing code? If the latter, could we just as easily have used fixed with simple arrays, and whatnot? Relevant to discussions I've had with @eerhardt ... we want to get rid of AlignedArray.
@eerhardteerhardtJun 26, 2018

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a few small benchmark performance tests I've run, AlignedArray performs worse than using a regular array.
@wschinwschinJun 26, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FFM's core computation is done by SSE code, which requires the memory blocks to be aligned. The main computation doesn't call any member functions of AlighedArray, we just use AlighedArray as something like new operator. Thus, the performance is possibly as good as regular array.
@eerhardteerhardtJun 26, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my investigation, I concluded that the performance penalty being paid was where we are moving the array elements around in memory to manually align it.
machinelearning/src/Microsoft.ML.CpuMath/AlignedArray.cs

Line 67
in
c023727
MoveData(ifltMin);
In my experience, doing this copying is worse performance than just using unaligned reads.
Another way to fix this issue is to use both a preamble and a postamble to get aligned. The idea is to do the operation one element at a time while you are not aligned at the beginning. Then once you hit the alignment, do the vectorization operation as usual. Then at the end, do one element at a time for the last elements that don't fill out a full section. #Pending
@sfilipisfilipiJun 26, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the note Eric. Added notes to Issue #146, to address this.
In reply to: 198231400 [](ancestors = 198231400)
 {
 Contracts.AssertValue(a);
 Contracts.Assert(a.Size > 0);
 return a.CbAlign == CbAlign;
 }

 private unsafe static float* Ptr(AlignedArray a, float* p)
 {
 Contracts.AssertValue(a);
 float* q = p + a.GetBase((long)p);
 Contracts.Assert(((long)q & (CbAlign - 1)) == 0);
 return q;
 }

 [DllImport(NativePath), SuppressUnmanagedCodeSecurity]
 public static extern void CalculateIntermediateVariablesNative(int fieldCount, int latentDim, int count, int* /*const*/ fieldIndices, int* /*const*/ featureIndices,
 float* /*const*/ featureValues, float* /*const*/ linearWeights, float* /*const*/ latentWeights, float* latentSum, float* response);
@Ivanidzo4kaIvanidzo4kaJun 20, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/const/ [](start = 51, length = 9)
are this comments here for a reason? #Resolved
@sfilipisfilipiJun 20, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wschin just remove them, or need to comment on them?
In reply to: 196861706 [](ancestors = 196861706)
@wschinwschinJun 20, 2018•

edited by sfilipi

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added those comments /*const*/ following another learner's implementation. It also uses C++ and C# together. The comments means "although they are not constant pointers, you should not touch their values." #Resolved
@sfilipisfilipiJun 21, 2018

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added this comment to the code.
In reply to: 196948210 [](ancestors = 196948210)

 [DllImport(NativePath), SuppressUnmanagedCodeSecurity]
 public static extern void CalculateGradientAndUpdateNative(float lambdaLinear, float lambdaLatent, float learningRate, int fieldCount, int latentDim, float weight,
 int count, int* /*const*/ fieldIndices, int* /*const*/ featureIndices, float* /*const*/ featureValues, float* /*const*/ latentSum, float slope,
 float* linearWeights, float* latentWeights, float* linearAccumulatedSquaredGrads, float* latentAccumulatedSquaredGrads);

 public static void CalculateIntermediateVariables(int fieldCount, int latentDim, int count, int[] fieldIndices, int[] featureIndices, float[] featureValues,
 float[] linearWeights, AlignedArray latentWeights, AlignedArray latentSum, ref float response)
 {
 Contracts.AssertNonEmpty(fieldIndices);
 Contracts.AssertNonEmpty(featureValues);
 Contracts.AssertNonEmpty(featureIndices);
 Contracts.AssertNonEmpty(linearWeights);
 Contracts.Assert(Compat(latentWeights));
 Contracts.Assert(Compat(latentSum));

 unsafe
 {
 fixed (int* pf = &fieldIndices[0])
 fixed (int* pi = &featureIndices[0])
 fixed (float* px = &featureValues[0])
 fixed (float* pw = &linearWeights[0])
 fixed (float* pv = &latentWeights.Items[0])
 fixed (float* pq = &latentSum.Items[0])
 fixed (float* pr = &response)
 CalculateIntermediateVariablesNative(fieldCount, latentDim, count, pf, pi, px, pw, Ptr(latentWeights, pv), Ptr(latentSum, pq), pr);
 }
 }

 public static void CalculateGradientAndUpdate(float lambdaLinear, float lambdaLatent, float learningRate, int fieldCount, int latentDim,
 float weight, int count, int[] fieldIndices, int[] featureIndices, float[] featureValues, AlignedArray latentSum, float slope,
 float[] linearWeights, AlignedArray latentWeights, float[] linearAccumulatedSquaredGrads, AlignedArray latentAccumulatedSquaredGrads)
 {
 Contracts.AssertNonEmpty(fieldIndices);
 Contracts.AssertNonEmpty(featureIndices);
 Contracts.AssertNonEmpty(featureValues);
 Contracts.Assert(Compat(latentSum));
 Contracts.AssertNonEmpty(linearWeights);
 Contracts.Assert(Compat(latentWeights));
 Contracts.AssertNonEmpty(linearAccumulatedSquaredGrads);
 Contracts.Assert(Compat(latentAccumulatedSquaredGrads));

 unsafe
 {
 fixed (int* pf = &fieldIndices[0])
 fixed (int* pi = &featureIndices[0])
 fixed (float* px = &featureValues[0])
 fixed (float* pq = &latentSum.Items[0])
 fixed (float* pw = &linearWeights[0])
 fixed (float* pv = &latentWeights.Items[0])
 fixed (float* phw = &linearAccumulatedSquaredGrads[0])
 fixed (float* phv = &latentAccumulatedSquaredGrads.Items[0])
 CalculateGradientAndUpdateNative(lambdaLinear, lambdaLatent, learningRate, fieldCount, latentDim, weight, count, pf, pi, px,
 Ptr(latentSum, pq), slope, pw, Ptr(latentWeights, pv), phw, Ptr(latentAccumulatedSquaredGrads, phv));
 }

 }
 }
}
Loading