Returning list of op names in get_graph_ops and extended error atoms to all TF error codes - #6

Merged
anshuman23 merged 3 commits into
masterfrom
dev
May 19, 2018
Merged

Returning list of op names in get_graph_ops and extended error atoms to all TF error codes#6
anshuman23 merged 3 commits into
masterfrom
dev

Conversation

@anshuman23

Copy link
Copy Markdown
Owner

So this PR covers the changes you had last requested @josevalim. Other than that I have just removed some unused code from before. Also ensured I am using enif_make_binnary wherever needed instead of enif_make_string:

  • Extending the error coverage: The C function error_to_string handles that now and an example of a faulty graph reading attempt would look like this:
iex(1)>graph=Tensorflex.read_graph"Makefile"{:error,:invalid_argument}
  • Returning list of all operations in graph instead of directly printing them: This has also been taken care of, and now the function is called get_graph_ops which returns a List of strings (op names in the graph). An example is as follows:
iex(1)>graph=Tensorflex.read_graph("classify_image_graph_def.pb")2018-05-1723:36:16.488469: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supportsinstructionsthatthisTensorFlowbinarywasnotcompiledtouse: SSE4.1 SSE4.2 AVX AVX2 FMA 2018-05-1723:36:16.774442: W tensorflow/core/framework/op_def_util.cc:334] OpBatchNormWithGlobalNormalization isdeprecated.It will cease to work inGraphDefversion9.Use tf.nn.batch_normalization().Successfullyimportedgraph#Reference<0.1610607974.1988231169.250293>iex(2)>op_list=Tensorflex.get_graph_opsgraph["softmax/biases","softmax/weights","pool_3/_reshape/shape","mixed_10/join/concat_dim","mixed_10/tower_2/conv/batchnorm/moving_variance","mixed_10/tower_2/conv/batchnorm/moving_mean","mixed_10/tower_2/conv/batchnorm/gamma","mixed_10/tower_2/conv/batchnorm/beta","mixed_10/tower_2/conv/conv2d_params","mixed_10/tower_1/mixed/conv_1/batchnorm/moving_variance","mixed_10/tower_1/mixed/conv_1/batchnorm/moving_mean","mixed_10/tower_1/mixed/conv_1/batchnorm/gamma","mixed_10/tower_1/mixed/conv_1/batchnorm/beta","mixed_10/tower_1/mixed/conv_1/conv2d_params","mixed_10/tower_1/mixed/conv/batchnorm/moving_variance","mixed_10/tower_1/mixed/conv/batchnorm/moving_mean","mixed_10/tower_1/mixed/conv/batchnorm/gamma","mixed_10/tower_1/mixed/conv/batchnorm/beta","mixed_10/tower_1/mixed/conv/conv2d_params","mixed_10/tower_1/conv_1/batchnorm/moving_variance","mixed_10/tower_1/conv_1/batchnorm/moving_mean","mixed_10/tower_1/conv_1/batchnorm/gamma","mixed_10/tower_1/conv_1/batchnorm/beta","mixed_10/tower_1/conv_1/conv2d_params","mixed_10/tower_1/conv/batchnorm/moving_variance","mixed_10/tower_1/conv/batchnorm/moving_mean","mixed_10/tower_1/conv/batchnorm/gamma","mixed_10/tower_1/conv/batchnorm/beta","mixed_10/tower_1/conv/conv2d_params","mixed_10/tower/mixed/conv_1/batchnorm/moving_variance","mixed_10/tower/mixed/conv_1/batchnorm/moving_mean","mixed_10/tower/mixed/conv_1/batchnorm/gamma","mixed_10/tower/mixed/conv_1/batchnorm/beta","mixed_10/tower/mixed/conv_1/conv2d_params","mixed_10/tower/mixed/conv/batchnorm/moving_variance","mixed_10/tower/mixed/conv/batchnorm/moving_mean","mixed_10/tower/mixed/conv/batchnorm/gamma","mixed_10/tower/mixed/conv/batchnorm/beta","mixed_10/tower/mixed/conv/conv2d_params","mixed_10/tower/conv/batchnorm/moving_variance","mixed_10/tower/conv/batchnorm/moving_mean","mixed_10/tower/conv/batchnorm/gamma","mixed_10/tower/conv/batchnorm/beta","mixed_10/tower/conv/conv2d_params","mixed_10/conv/batchnorm/moving_variance","mixed_10/conv/batchnorm/moving_mean","mixed_10/conv/batchnorm/gamma","mixed_10/conv/batchnorm/beta","mixed_10/conv/conv2d_params","mixed_9/join/concat_dim",...]

Comment threadc_src/Tensorflex.c Outdated
break;
case TF_DATA_LOSS: strcpy(error,"data_loss");
break;
default: strcpy(error,"unlisted_code");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we could have each of those call enif_make_atom so we don't have to allocate the string in the first place, just to convert it to an atom?

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed this in the latest commit. I was needlessly allocating a string for this. Renamed the function to error_to_atom now.

Comment threadc_src/Tensorflex.c Outdated
char op_name[BASE_STRING_LENGTH];
for(int i=0; i<n_ops; i++) {
op_temp = TF_GraphNextOperation(*graph, &pos);
strcpy(op_name, (char*) TF_OperationName(op_temp));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to copy the string here? Given that we are already copying it later to the binary with memcpy?

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, this wasn't needed and I have fixed this in the latest commit. I apologize for the poorly written code here.

@josevalim

Copy link
Copy Markdown
Contributor

Comments added. Other than that, it looks great to go! 👍

What next steps do you have in mind? :)

@anshuman23

Copy link
Copy Markdown
OwnerAuthor

Thanks for the feedback @josevalim!

I have been thinking about the next steps, and the logical progression of adding functions should eventually get to the point where we can run the loaded graph against our own inputs and generate prediction outputs. For this to work, a lot of functions need to be added first: particularly ones that will allow us to read Tensors supplied by the user. I am thinking over how I would accomplish this and then I will submit a PR for review. These TF_Tensor functions are very important and they would have to encompass a wide variety of inputs.

I might also keep importing graph based functions from Tensorflow as and when required (like the operation list function get_graph_ops) After that, I think some more graph based functions will be needed that will use both the tensor functionality and the graph functions. Finally, we will need to add support for creating and running Sessions.

@anshuman23
anshuman23 merged commit 2398996 into masterMay 19, 2018
@josevalim

Copy link
Copy Markdown
Contributor

@anshuman23 agreed! ❤️

Let's go with this plan and continue exploring the API and the "user story" but note that at some point we will have to "sit down" and write documentation and tests.

Also, can you please send an e-mail to the beamcommunity mailing list with your progress so far and your plans for the next week? What is coming next can be a very quick digest, like this one that you just sent!

@anshuman23

Copy link
Copy Markdown
OwnerAuthor

Thanks @josevalim, and I completely agree on taking time out to write documentation and tests. I will incorporate that into my plans as well. :)

Yes, I'll send that email right away. Thanks again for all the help!

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@anshuman23@josevalim
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Returning list of op names in get_graph_ops and extended error atoms to all TF error codes - #6

Merged
anshuman23 merged 3 commits into
masterfrom
dev
May 19, 2018
Merged

Returning list of op names in get_graph_ops and extended error atoms to all TF error codes#6
anshuman23 merged 3 commits into
masterfrom
dev

Conversation

@anshuman23

Copy link
Copy Markdown
Owner

So this PR covers the changes you had last requested @josevalim. Other than that I have just removed some unused code from before. Also ensured I am using enif_make_binnary wherever needed instead of enif_make_string:

  • Extending the error coverage: The C function error_to_string handles that now and an example of a faulty graph reading attempt would look like this:
iex(1)>graph=Tensorflex.read_graph"Makefile"{:error,:invalid_argument}
  • Returning list of all operations in graph instead of directly printing them: This has also been taken care of, and now the function is called get_graph_ops which returns a List of strings (op names in the graph). An example is as follows:
iex(1)>graph=Tensorflex.read_graph("classify_image_graph_def.pb")2018-05-1723:36:16.488469: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supportsinstructionsthatthisTensorFlowbinarywasnotcompiledtouse: SSE4.1 SSE4.2 AVX AVX2 FMA 2018-05-1723:36:16.774442: W tensorflow/core/framework/op_def_util.cc:334] OpBatchNormWithGlobalNormalization isdeprecated.It will cease to work inGraphDefversion9.Use tf.nn.batch_normalization().Successfullyimportedgraph#Reference<0.1610607974.1988231169.250293>iex(2)>op_list=Tensorflex.get_graph_opsgraph["softmax/biases","softmax/weights","pool_3/_reshape/shape","mixed_10/join/concat_dim","mixed_10/tower_2/conv/batchnorm/moving_variance","mixed_10/tower_2/conv/batchnorm/moving_mean","mixed_10/tower_2/conv/batchnorm/gamma","mixed_10/tower_2/conv/batchnorm/beta","mixed_10/tower_2/conv/conv2d_params","mixed_10/tower_1/mixed/conv_1/batchnorm/moving_variance","mixed_10/tower_1/mixed/conv_1/batchnorm/moving_mean","mixed_10/tower_1/mixed/conv_1/batchnorm/gamma","mixed_10/tower_1/mixed/conv_1/batchnorm/beta","mixed_10/tower_1/mixed/conv_1/conv2d_params","mixed_10/tower_1/mixed/conv/batchnorm/moving_variance","mixed_10/tower_1/mixed/conv/batchnorm/moving_mean","mixed_10/tower_1/mixed/conv/batchnorm/gamma","mixed_10/tower_1/mixed/conv/batchnorm/beta","mixed_10/tower_1/mixed/conv/conv2d_params","mixed_10/tower_1/conv_1/batchnorm/moving_variance","mixed_10/tower_1/conv_1/batchnorm/moving_mean","mixed_10/tower_1/conv_1/batchnorm/gamma","mixed_10/tower_1/conv_1/batchnorm/beta","mixed_10/tower_1/conv_1/conv2d_params","mixed_10/tower_1/conv/batchnorm/moving_variance","mixed_10/tower_1/conv/batchnorm/moving_mean","mixed_10/tower_1/conv/batchnorm/gamma","mixed_10/tower_1/conv/batchnorm/beta","mixed_10/tower_1/conv/conv2d_params","mixed_10/tower/mixed/conv_1/batchnorm/moving_variance","mixed_10/tower/mixed/conv_1/batchnorm/moving_mean","mixed_10/tower/mixed/conv_1/batchnorm/gamma","mixed_10/tower/mixed/conv_1/batchnorm/beta","mixed_10/tower/mixed/conv_1/conv2d_params","mixed_10/tower/mixed/conv/batchnorm/moving_variance","mixed_10/tower/mixed/conv/batchnorm/moving_mean","mixed_10/tower/mixed/conv/batchnorm/gamma","mixed_10/tower/mixed/conv/batchnorm/beta","mixed_10/tower/mixed/conv/conv2d_params","mixed_10/tower/conv/batchnorm/moving_variance","mixed_10/tower/conv/batchnorm/moving_mean","mixed_10/tower/conv/batchnorm/gamma","mixed_10/tower/conv/batchnorm/beta","mixed_10/tower/conv/conv2d_params","mixed_10/conv/batchnorm/moving_variance","mixed_10/conv/batchnorm/moving_mean","mixed_10/conv/batchnorm/gamma","mixed_10/conv/batchnorm/beta","mixed_10/conv/conv2d_params","mixed_9/join/concat_dim",...]

Comment threadc_src/Tensorflex.c Outdated
break;
case TF_DATA_LOSS: strcpy(error,"data_loss");
break;
default: strcpy(error,"unlisted_code");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we could have each of those call enif_make_atom so we don't have to allocate the string in the first place, just to convert it to an atom?

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed this in the latest commit. I was needlessly allocating a string for this. Renamed the function to error_to_atom now.

Comment threadc_src/Tensorflex.c Outdated
char op_name[BASE_STRING_LENGTH];
for(int i=0; i<n_ops; i++) {
op_temp = TF_GraphNextOperation(*graph, &pos);
strcpy(op_name, (char*) TF_OperationName(op_temp));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to copy the string here? Given that we are already copying it later to the binary with memcpy?

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, this wasn't needed and I have fixed this in the latest commit. I apologize for the poorly written code here.

@josevalim

Copy link
Copy Markdown
Contributor

Comments added. Other than that, it looks great to go! 👍

What next steps do you have in mind? :)

@anshuman23

Copy link
Copy Markdown
OwnerAuthor

Thanks for the feedback @josevalim!

I have been thinking about the next steps, and the logical progression of adding functions should eventually get to the point where we can run the loaded graph against our own inputs and generate prediction outputs. For this to work, a lot of functions need to be added first: particularly ones that will allow us to read Tensors supplied by the user. I am thinking over how I would accomplish this and then I will submit a PR for review. These TF_Tensor functions are very important and they would have to encompass a wide variety of inputs.

I might also keep importing graph based functions from Tensorflow as and when required (like the operation list function get_graph_ops) After that, I think some more graph based functions will be needed that will use both the tensor functionality and the graph functions. Finally, we will need to add support for creating and running Sessions.

@anshuman23
anshuman23 merged commit 2398996 into masterMay 19, 2018
@josevalim

Copy link
Copy Markdown
Contributor

@anshuman23 agreed! ❤️

Let's go with this plan and continue exploring the API and the "user story" but note that at some point we will have to "sit down" and write documentation and tests.

Also, can you please send an e-mail to the beamcommunity mailing list with your progress so far and your plans for the next week? What is coming next can be a very quick digest, like this one that you just sent!

@anshuman23

Copy link
Copy Markdown
OwnerAuthor

Thanks @josevalim, and I completely agree on taking time out to write documentation and tests. I will incorporate that into my plans as well. :)

Yes, I'll send that email right away. Thanks again for all the help!

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@anshuman23@josevalim
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Returning list of op names in get_graph_ops and extended error atoms to all TF error codes - #6

Merged
anshuman23 merged 3 commits into
masterfrom
dev
May 19, 2018
Merged

Returning list of op names in get_graph_ops and extended error atoms to all TF error codes#6
anshuman23 merged 3 commits into
masterfrom
dev

Conversation

@anshuman23

Copy link
Copy Markdown
Owner

So this PR covers the changes you had last requested @josevalim. Other than that I have just removed some unused code from before. Also ensured I am using enif_make_binnary wherever needed instead of enif_make_string:

  • Extending the error coverage: The C function error_to_string handles that now and an example of a faulty graph reading attempt would look like this:
iex(1)>graph=Tensorflex.read_graph"Makefile"{:error,:invalid_argument}
  • Returning list of all operations in graph instead of directly printing them: This has also been taken care of, and now the function is called get_graph_ops which returns a List of strings (op names in the graph). An example is as follows:
iex(1)>graph=Tensorflex.read_graph("classify_image_graph_def.pb")2018-05-1723:36:16.488469: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supportsinstructionsthatthisTensorFlowbinarywasnotcompiledtouse: SSE4.1 SSE4.2 AVX AVX2 FMA 2018-05-1723:36:16.774442: W tensorflow/core/framework/op_def_util.cc:334] OpBatchNormWithGlobalNormalization isdeprecated.It will cease to work inGraphDefversion9.Use tf.nn.batch_normalization().Successfullyimportedgraph#Reference<0.1610607974.1988231169.250293>iex(2)>op_list=Tensorflex.get_graph_opsgraph["softmax/biases","softmax/weights","pool_3/_reshape/shape","mixed_10/join/concat_dim","mixed_10/tower_2/conv/batchnorm/moving_variance","mixed_10/tower_2/conv/batchnorm/moving_mean","mixed_10/tower_2/conv/batchnorm/gamma","mixed_10/tower_2/conv/batchnorm/beta","mixed_10/tower_2/conv/conv2d_params","mixed_10/tower_1/mixed/conv_1/batchnorm/moving_variance","mixed_10/tower_1/mixed/conv_1/batchnorm/moving_mean","mixed_10/tower_1/mixed/conv_1/batchnorm/gamma","mixed_10/tower_1/mixed/conv_1/batchnorm/beta","mixed_10/tower_1/mixed/conv_1/conv2d_params","mixed_10/tower_1/mixed/conv/batchnorm/moving_variance","mixed_10/tower_1/mixed/conv/batchnorm/moving_mean","mixed_10/tower_1/mixed/conv/batchnorm/gamma","mixed_10/tower_1/mixed/conv/batchnorm/beta","mixed_10/tower_1/mixed/conv/conv2d_params","mixed_10/tower_1/conv_1/batchnorm/moving_variance","mixed_10/tower_1/conv_1/batchnorm/moving_mean","mixed_10/tower_1/conv_1/batchnorm/gamma","mixed_10/tower_1/conv_1/batchnorm/beta","mixed_10/tower_1/conv_1/conv2d_params","mixed_10/tower_1/conv/batchnorm/moving_variance","mixed_10/tower_1/conv/batchnorm/moving_mean","mixed_10/tower_1/conv/batchnorm/gamma","mixed_10/tower_1/conv/batchnorm/beta","mixed_10/tower_1/conv/conv2d_params","mixed_10/tower/mixed/conv_1/batchnorm/moving_variance","mixed_10/tower/mixed/conv_1/batchnorm/moving_mean","mixed_10/tower/mixed/conv_1/batchnorm/gamma","mixed_10/tower/mixed/conv_1/batchnorm/beta","mixed_10/tower/mixed/conv_1/conv2d_params","mixed_10/tower/mixed/conv/batchnorm/moving_variance","mixed_10/tower/mixed/conv/batchnorm/moving_mean","mixed_10/tower/mixed/conv/batchnorm/gamma","mixed_10/tower/mixed/conv/batchnorm/beta","mixed_10/tower/mixed/conv/conv2d_params","mixed_10/tower/conv/batchnorm/moving_variance","mixed_10/tower/conv/batchnorm/moving_mean","mixed_10/tower/conv/batchnorm/gamma","mixed_10/tower/conv/batchnorm/beta","mixed_10/tower/conv/conv2d_params","mixed_10/conv/batchnorm/moving_variance","mixed_10/conv/batchnorm/moving_mean","mixed_10/conv/batchnorm/gamma","mixed_10/conv/batchnorm/beta","mixed_10/conv/conv2d_params","mixed_9/join/concat_dim",...]

Comment threadc_src/Tensorflex.c Outdated
break;
case TF_DATA_LOSS: strcpy(error,"data_loss");
break;
default: strcpy(error,"unlisted_code");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we could have each of those call enif_make_atom so we don't have to allocate the string in the first place, just to convert it to an atom?

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed this in the latest commit. I was needlessly allocating a string for this. Renamed the function to error_to_atom now.

Comment threadc_src/Tensorflex.c Outdated
char op_name[BASE_STRING_LENGTH];
for(int i=0; i<n_ops; i++) {
op_temp = TF_GraphNextOperation(*graph, &pos);
strcpy(op_name, (char*) TF_OperationName(op_temp));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to copy the string here? Given that we are already copying it later to the binary with memcpy?

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, this wasn't needed and I have fixed this in the latest commit. I apologize for the poorly written code here.

@josevalim

Copy link
Copy Markdown
Contributor

Comments added. Other than that, it looks great to go! 👍

What next steps do you have in mind? :)

@anshuman23

Copy link
Copy Markdown
OwnerAuthor

Thanks for the feedback @josevalim!

I have been thinking about the next steps, and the logical progression of adding functions should eventually get to the point where we can run the loaded graph against our own inputs and generate prediction outputs. For this to work, a lot of functions need to be added first: particularly ones that will allow us to read Tensors supplied by the user. I am thinking over how I would accomplish this and then I will submit a PR for review. These TF_Tensor functions are very important and they would have to encompass a wide variety of inputs.

I might also keep importing graph based functions from Tensorflow as and when required (like the operation list function get_graph_ops) After that, I think some more graph based functions will be needed that will use both the tensor functionality and the graph functions. Finally, we will need to add support for creating and running Sessions.

@anshuman23
anshuman23 merged commit 2398996 into masterMay 19, 2018
@josevalim

Copy link
Copy Markdown
Contributor

@anshuman23 agreed! ❤️

Let's go with this plan and continue exploring the API and the "user story" but note that at some point we will have to "sit down" and write documentation and tests.

Also, can you please send an e-mail to the beamcommunity mailing list with your progress so far and your plans for the next week? What is coming next can be a very quick digest, like this one that you just sent!

@anshuman23

Copy link
Copy Markdown
OwnerAuthor

Thanks @josevalim, and I completely agree on taking time out to write documentation and tests. I will incorporate that into my plans as well. :)

Yes, I'll send that email right away. Thanks again for all the help!

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@anshuman23@josevalim
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Returning list of op names in get_graph_ops and extended error atoms to all TF error codes - #6

Merged
anshuman23 merged 3 commits into
masterfrom
dev
May 19, 2018
Merged

Returning list of op names in get_graph_ops and extended error atoms to all TF error codes#6
anshuman23 merged 3 commits into
masterfrom
dev

Conversation

@anshuman23

Copy link
Copy Markdown
Owner

So this PR covers the changes you had last requested @josevalim. Other than that I have just removed some unused code from before. Also ensured I am using enif_make_binnary wherever needed instead of enif_make_string:

  • Extending the error coverage: The C function error_to_string handles that now and an example of a faulty graph reading attempt would look like this:
iex(1)>graph=Tensorflex.read_graph"Makefile"{:error,:invalid_argument}
  • Returning list of all operations in graph instead of directly printing them: This has also been taken care of, and now the function is called get_graph_ops which returns a List of strings (op names in the graph). An example is as follows:
iex(1)>graph=Tensorflex.read_graph("classify_image_graph_def.pb")2018-05-1723:36:16.488469: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supportsinstructionsthatthisTensorFlowbinarywasnotcompiledtouse: SSE4.1 SSE4.2 AVX AVX2 FMA 2018-05-1723:36:16.774442: W tensorflow/core/framework/op_def_util.cc:334] OpBatchNormWithGlobalNormalization isdeprecated.It will cease to work inGraphDefversion9.Use tf.nn.batch_normalization().Successfullyimportedgraph#Reference<0.1610607974.1988231169.250293>iex(2)>op_list=Tensorflex.get_graph_opsgraph["softmax/biases","softmax/weights","pool_3/_reshape/shape","mixed_10/join/concat_dim","mixed_10/tower_2/conv/batchnorm/moving_variance","mixed_10/tower_2/conv/batchnorm/moving_mean","mixed_10/tower_2/conv/batchnorm/gamma","mixed_10/tower_2/conv/batchnorm/beta","mixed_10/tower_2/conv/conv2d_params","mixed_10/tower_1/mixed/conv_1/batchnorm/moving_variance","mixed_10/tower_1/mixed/conv_1/batchnorm/moving_mean","mixed_10/tower_1/mixed/conv_1/batchnorm/gamma","mixed_10/tower_1/mixed/conv_1/batchnorm/beta","mixed_10/tower_1/mixed/conv_1/conv2d_params","mixed_10/tower_1/mixed/conv/batchnorm/moving_variance","mixed_10/tower_1/mixed/conv/batchnorm/moving_mean","mixed_10/tower_1/mixed/conv/batchnorm/gamma","mixed_10/tower_1/mixed/conv/batchnorm/beta","mixed_10/tower_1/mixed/conv/conv2d_params","mixed_10/tower_1/conv_1/batchnorm/moving_variance","mixed_10/tower_1/conv_1/batchnorm/moving_mean","mixed_10/tower_1/conv_1/batchnorm/gamma","mixed_10/tower_1/conv_1/batchnorm/beta","mixed_10/tower_1/conv_1/conv2d_params","mixed_10/tower_1/conv/batchnorm/moving_variance","mixed_10/tower_1/conv/batchnorm/moving_mean","mixed_10/tower_1/conv/batchnorm/gamma","mixed_10/tower_1/conv/batchnorm/beta","mixed_10/tower_1/conv/conv2d_params","mixed_10/tower/mixed/conv_1/batchnorm/moving_variance","mixed_10/tower/mixed/conv_1/batchnorm/moving_mean","mixed_10/tower/mixed/conv_1/batchnorm/gamma","mixed_10/tower/mixed/conv_1/batchnorm/beta","mixed_10/tower/mixed/conv_1/conv2d_params","mixed_10/tower/mixed/conv/batchnorm/moving_variance","mixed_10/tower/mixed/conv/batchnorm/moving_mean","mixed_10/tower/mixed/conv/batchnorm/gamma","mixed_10/tower/mixed/conv/batchnorm/beta","mixed_10/tower/mixed/conv/conv2d_params","mixed_10/tower/conv/batchnorm/moving_variance","mixed_10/tower/conv/batchnorm/moving_mean","mixed_10/tower/conv/batchnorm/gamma","mixed_10/tower/conv/batchnorm/beta","mixed_10/tower/conv/conv2d_params","mixed_10/conv/batchnorm/moving_variance","mixed_10/conv/batchnorm/moving_mean","mixed_10/conv/batchnorm/gamma","mixed_10/conv/batchnorm/beta","mixed_10/conv/conv2d_params","mixed_9/join/concat_dim",...]

Comment threadc_src/Tensorflex.c Outdated
break;
case TF_DATA_LOSS: strcpy(error,"data_loss");
break;
default: strcpy(error,"unlisted_code");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we could have each of those call enif_make_atom so we don't have to allocate the string in the first place, just to convert it to an atom?

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed this in the latest commit. I was needlessly allocating a string for this. Renamed the function to error_to_atom now.

Comment threadc_src/Tensorflex.c Outdated
char op_name[BASE_STRING_LENGTH];
for(int i=0; i<n_ops; i++) {
op_temp = TF_GraphNextOperation(*graph, &pos);
strcpy(op_name, (char*) TF_OperationName(op_temp));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to copy the string here? Given that we are already copying it later to the binary with memcpy?

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, this wasn't needed and I have fixed this in the latest commit. I apologize for the poorly written code here.

@josevalim

Copy link
Copy Markdown
Contributor

Comments added. Other than that, it looks great to go! 👍

What next steps do you have in mind? :)

@anshuman23

Copy link
Copy Markdown
OwnerAuthor

Thanks for the feedback @josevalim!

I have been thinking about the next steps, and the logical progression of adding functions should eventually get to the point where we can run the loaded graph against our own inputs and generate prediction outputs. For this to work, a lot of functions need to be added first: particularly ones that will allow us to read Tensors supplied by the user. I am thinking over how I would accomplish this and then I will submit a PR for review. These TF_Tensor functions are very important and they would have to encompass a wide variety of inputs.

I might also keep importing graph based functions from Tensorflow as and when required (like the operation list function get_graph_ops) After that, I think some more graph based functions will be needed that will use both the tensor functionality and the graph functions. Finally, we will need to add support for creating and running Sessions.

@anshuman23
anshuman23 merged commit 2398996 into masterMay 19, 2018
@josevalim

Copy link
Copy Markdown
Contributor

@anshuman23 agreed! ❤️

Let's go with this plan and continue exploring the API and the "user story" but note that at some point we will have to "sit down" and write documentation and tests.

Also, can you please send an e-mail to the beamcommunity mailing list with your progress so far and your plans for the next week? What is coming next can be a very quick digest, like this one that you just sent!

@anshuman23

Copy link
Copy Markdown
OwnerAuthor

Thanks @josevalim, and I completely agree on taking time out to write documentation and tests. I will incorporate that into my plans as well. :)

Yes, I'll send that email right away. Thanks again for all the help!

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@anshuman23@josevalim
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Returning list of op names in get_graph_ops and extended error atoms to all TF error codes - #6

Merged
anshuman23 merged 3 commits into
masterfrom
dev
May 19, 2018
Merged

Returning list of op names in get_graph_ops and extended error atoms to all TF error codes#6
anshuman23 merged 3 commits into
masterfrom
dev

Conversation

@anshuman23

Copy link
Copy Markdown
Owner

So this PR covers the changes you had last requested @josevalim. Other than that I have just removed some unused code from before. Also ensured I am using enif_make_binnary wherever needed instead of enif_make_string:

  • Extending the error coverage: The C function error_to_string handles that now and an example of a faulty graph reading attempt would look like this:
iex(1)>graph=Tensorflex.read_graph"Makefile"{:error,:invalid_argument}
  • Returning list of all operations in graph instead of directly printing them: This has also been taken care of, and now the function is called get_graph_ops which returns a List of strings (op names in the graph). An example is as follows:
iex(1)>graph=Tensorflex.read_graph("classify_image_graph_def.pb")2018-05-1723:36:16.488469: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supportsinstructionsthatthisTensorFlowbinarywasnotcompiledtouse: SSE4.1 SSE4.2 AVX AVX2 FMA 2018-05-1723:36:16.774442: W tensorflow/core/framework/op_def_util.cc:334] OpBatchNormWithGlobalNormalization isdeprecated.It will cease to work inGraphDefversion9.Use tf.nn.batch_normalization().Successfullyimportedgraph#Reference<0.1610607974.1988231169.250293>iex(2)>op_list=Tensorflex.get_graph_opsgraph["softmax/biases","softmax/weights","pool_3/_reshape/shape","mixed_10/join/concat_dim","mixed_10/tower_2/conv/batchnorm/moving_variance","mixed_10/tower_2/conv/batchnorm/moving_mean","mixed_10/tower_2/conv/batchnorm/gamma","mixed_10/tower_2/conv/batchnorm/beta","mixed_10/tower_2/conv/conv2d_params","mixed_10/tower_1/mixed/conv_1/batchnorm/moving_variance","mixed_10/tower_1/mixed/conv_1/batchnorm/moving_mean","mixed_10/tower_1/mixed/conv_1/batchnorm/gamma","mixed_10/tower_1/mixed/conv_1/batchnorm/beta","mixed_10/tower_1/mixed/conv_1/conv2d_params","mixed_10/tower_1/mixed/conv/batchnorm/moving_variance","mixed_10/tower_1/mixed/conv/batchnorm/moving_mean","mixed_10/tower_1/mixed/conv/batchnorm/gamma","mixed_10/tower_1/mixed/conv/batchnorm/beta","mixed_10/tower_1/mixed/conv/conv2d_params","mixed_10/tower_1/conv_1/batchnorm/moving_variance","mixed_10/tower_1/conv_1/batchnorm/moving_mean","mixed_10/tower_1/conv_1/batchnorm/gamma","mixed_10/tower_1/conv_1/batchnorm/beta","mixed_10/tower_1/conv_1/conv2d_params","mixed_10/tower_1/conv/batchnorm/moving_variance","mixed_10/tower_1/conv/batchnorm/moving_mean","mixed_10/tower_1/conv/batchnorm/gamma","mixed_10/tower_1/conv/batchnorm/beta","mixed_10/tower_1/conv/conv2d_params","mixed_10/tower/mixed/conv_1/batchnorm/moving_variance","mixed_10/tower/mixed/conv_1/batchnorm/moving_mean","mixed_10/tower/mixed/conv_1/batchnorm/gamma","mixed_10/tower/mixed/conv_1/batchnorm/beta","mixed_10/tower/mixed/conv_1/conv2d_params","mixed_10/tower/mixed/conv/batchnorm/moving_variance","mixed_10/tower/mixed/conv/batchnorm/moving_mean","mixed_10/tower/mixed/conv/batchnorm/gamma","mixed_10/tower/mixed/conv/batchnorm/beta","mixed_10/tower/mixed/conv/conv2d_params","mixed_10/tower/conv/batchnorm/moving_variance","mixed_10/tower/conv/batchnorm/moving_mean","mixed_10/tower/conv/batchnorm/gamma","mixed_10/tower/conv/batchnorm/beta","mixed_10/tower/conv/conv2d_params","mixed_10/conv/batchnorm/moving_variance","mixed_10/conv/batchnorm/moving_mean","mixed_10/conv/batchnorm/gamma","mixed_10/conv/batchnorm/beta","mixed_10/conv/conv2d_params","mixed_9/join/concat_dim",...]

Comment threadc_src/Tensorflex.c Outdated
break;
case TF_DATA_LOSS: strcpy(error,"data_loss");
break;
default: strcpy(error,"unlisted_code");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we could have each of those call enif_make_atom so we don't have to allocate the string in the first place, just to convert it to an atom?

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed this in the latest commit. I was needlessly allocating a string for this. Renamed the function to error_to_atom now.

Comment threadc_src/Tensorflex.c Outdated
char op_name[BASE_STRING_LENGTH];
for(int i=0; i<n_ops; i++) {
op_temp = TF_GraphNextOperation(*graph, &pos);
strcpy(op_name, (char*) TF_OperationName(op_temp));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to copy the string here? Given that we are already copying it later to the binary with memcpy?

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, this wasn't needed and I have fixed this in the latest commit. I apologize for the poorly written code here.

@josevalim

Copy link
Copy Markdown
Contributor

Comments added. Other than that, it looks great to go! 👍

What next steps do you have in mind? :)

@anshuman23

Copy link
Copy Markdown
OwnerAuthor

Thanks for the feedback @josevalim!

I have been thinking about the next steps, and the logical progression of adding functions should eventually get to the point where we can run the loaded graph against our own inputs and generate prediction outputs. For this to work, a lot of functions need to be added first: particularly ones that will allow us to read Tensors supplied by the user. I am thinking over how I would accomplish this and then I will submit a PR for review. These TF_Tensor functions are very important and they would have to encompass a wide variety of inputs.

I might also keep importing graph based functions from Tensorflow as and when required (like the operation list function get_graph_ops) After that, I think some more graph based functions will be needed that will use both the tensor functionality and the graph functions. Finally, we will need to add support for creating and running Sessions.

@anshuman23
anshuman23 merged commit 2398996 into masterMay 19, 2018
@josevalim

Copy link
Copy Markdown
Contributor

@anshuman23 agreed! ❤️

Let's go with this plan and continue exploring the API and the "user story" but note that at some point we will have to "sit down" and write documentation and tests.

Also, can you please send an e-mail to the beamcommunity mailing list with your progress so far and your plans for the next week? What is coming next can be a very quick digest, like this one that you just sent!

@anshuman23

Copy link
Copy Markdown
OwnerAuthor

Thanks @josevalim, and I completely agree on taking time out to write documentation and tests. I will incorporate that into my plans as well. :)

Yes, I'll send that email right away. Thanks again for all the help!

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@anshuman23@josevalim
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Returning list of op names in get_graph_ops and extended error atoms to all TF error codes - #6

Merged
anshuman23 merged 3 commits into
masterfrom
dev
May 19, 2018
Merged

Returning list of op names in get_graph_ops and extended error atoms to all TF error codes#6
anshuman23 merged 3 commits into
masterfrom
dev

Conversation

@anshuman23

Copy link
Copy Markdown
Owner

So this PR covers the changes you had last requested @josevalim. Other than that I have just removed some unused code from before. Also ensured I am using enif_make_binnary wherever needed instead of enif_make_string:

  • Extending the error coverage: The C function error_to_string handles that now and an example of a faulty graph reading attempt would look like this:
iex(1)>graph=Tensorflex.read_graph"Makefile"{:error,:invalid_argument}
  • Returning list of all operations in graph instead of directly printing them: This has also been taken care of, and now the function is called get_graph_ops which returns a List of strings (op names in the graph). An example is as follows:
iex(1)>graph=Tensorflex.read_graph("classify_image_graph_def.pb")2018-05-1723:36:16.488469: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supportsinstructionsthatthisTensorFlowbinarywasnotcompiledtouse: SSE4.1 SSE4.2 AVX AVX2 FMA 2018-05-1723:36:16.774442: W tensorflow/core/framework/op_def_util.cc:334] OpBatchNormWithGlobalNormalization isdeprecated.It will cease to work inGraphDefversion9.Use tf.nn.batch_normalization().Successfullyimportedgraph#Reference<0.1610607974.1988231169.250293>iex(2)>op_list=Tensorflex.get_graph_opsgraph["softmax/biases","softmax/weights","pool_3/_reshape/shape","mixed_10/join/concat_dim","mixed_10/tower_2/conv/batchnorm/moving_variance","mixed_10/tower_2/conv/batchnorm/moving_mean","mixed_10/tower_2/conv/batchnorm/gamma","mixed_10/tower_2/conv/batchnorm/beta","mixed_10/tower_2/conv/conv2d_params","mixed_10/tower_1/mixed/conv_1/batchnorm/moving_variance","mixed_10/tower_1/mixed/conv_1/batchnorm/moving_mean","mixed_10/tower_1/mixed/conv_1/batchnorm/gamma","mixed_10/tower_1/mixed/conv_1/batchnorm/beta","mixed_10/tower_1/mixed/conv_1/conv2d_params","mixed_10/tower_1/mixed/conv/batchnorm/moving_variance","mixed_10/tower_1/mixed/conv/batchnorm/moving_mean","mixed_10/tower_1/mixed/conv/batchnorm/gamma","mixed_10/tower_1/mixed/conv/batchnorm/beta","mixed_10/tower_1/mixed/conv/conv2d_params","mixed_10/tower_1/conv_1/batchnorm/moving_variance","mixed_10/tower_1/conv_1/batchnorm/moving_mean","mixed_10/tower_1/conv_1/batchnorm/gamma","mixed_10/tower_1/conv_1/batchnorm/beta","mixed_10/tower_1/conv_1/conv2d_params","mixed_10/tower_1/conv/batchnorm/moving_variance","mixed_10/tower_1/conv/batchnorm/moving_mean","mixed_10/tower_1/conv/batchnorm/gamma","mixed_10/tower_1/conv/batchnorm/beta","mixed_10/tower_1/conv/conv2d_params","mixed_10/tower/mixed/conv_1/batchnorm/moving_variance","mixed_10/tower/mixed/conv_1/batchnorm/moving_mean","mixed_10/tower/mixed/conv_1/batchnorm/gamma","mixed_10/tower/mixed/conv_1/batchnorm/beta","mixed_10/tower/mixed/conv_1/conv2d_params","mixed_10/tower/mixed/conv/batchnorm/moving_variance","mixed_10/tower/mixed/conv/batchnorm/moving_mean","mixed_10/tower/mixed/conv/batchnorm/gamma","mixed_10/tower/mixed/conv/batchnorm/beta","mixed_10/tower/mixed/conv/conv2d_params","mixed_10/tower/conv/batchnorm/moving_variance","mixed_10/tower/conv/batchnorm/moving_mean","mixed_10/tower/conv/batchnorm/gamma","mixed_10/tower/conv/batchnorm/beta","mixed_10/tower/conv/conv2d_params","mixed_10/conv/batchnorm/moving_variance","mixed_10/conv/batchnorm/moving_mean","mixed_10/conv/batchnorm/gamma","mixed_10/conv/batchnorm/beta","mixed_10/conv/conv2d_params","mixed_9/join/concat_dim",...]

Comment threadc_src/Tensorflex.c Outdated
break;
case TF_DATA_LOSS: strcpy(error,"data_loss");
break;
default: strcpy(error,"unlisted_code");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we could have each of those call enif_make_atom so we don't have to allocate the string in the first place, just to convert it to an atom?

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed this in the latest commit. I was needlessly allocating a string for this. Renamed the function to error_to_atom now.

Comment threadc_src/Tensorflex.c Outdated
char op_name[BASE_STRING_LENGTH];
for(int i=0; i<n_ops; i++) {
op_temp = TF_GraphNextOperation(*graph, &pos);
strcpy(op_name, (char*) TF_OperationName(op_temp));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to copy the string here? Given that we are already copying it later to the binary with memcpy?

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, this wasn't needed and I have fixed this in the latest commit. I apologize for the poorly written code here.

@josevalim

Copy link
Copy Markdown
Contributor

Comments added. Other than that, it looks great to go! 👍

What next steps do you have in mind? :)

@anshuman23

Copy link
Copy Markdown
OwnerAuthor

Thanks for the feedback @josevalim!

I have been thinking about the next steps, and the logical progression of adding functions should eventually get to the point where we can run the loaded graph against our own inputs and generate prediction outputs. For this to work, a lot of functions need to be added first: particularly ones that will allow us to read Tensors supplied by the user. I am thinking over how I would accomplish this and then I will submit a PR for review. These TF_Tensor functions are very important and they would have to encompass a wide variety of inputs.

I might also keep importing graph based functions from Tensorflow as and when required (like the operation list function get_graph_ops) After that, I think some more graph based functions will be needed that will use both the tensor functionality and the graph functions. Finally, we will need to add support for creating and running Sessions.

@anshuman23
anshuman23 merged commit 2398996 into masterMay 19, 2018
@josevalim

Copy link
Copy Markdown
Contributor

@anshuman23 agreed! ❤️

Let's go with this plan and continue exploring the API and the "user story" but note that at some point we will have to "sit down" and write documentation and tests.

Also, can you please send an e-mail to the beamcommunity mailing list with your progress so far and your plans for the next week? What is coming next can be a very quick digest, like this one that you just sent!

@anshuman23

Copy link
Copy Markdown
OwnerAuthor

Thanks @josevalim, and I completely agree on taking time out to write documentation and tests. I will incorporate that into my plans as well. :)

Yes, I'll send that email right away. Thanks again for all the help!

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@anshuman23@josevalim
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Returning list of op names in get_graph_ops and extended error atoms to all TF error codes - #6

Merged
anshuman23 merged 3 commits into
masterfrom
dev
May 19, 2018
Merged

Returning list of op names in get_graph_ops and extended error atoms to all TF error codes#6
anshuman23 merged 3 commits into
masterfrom
dev

Conversation

@anshuman23

Copy link
Copy Markdown
Owner

So this PR covers the changes you had last requested @josevalim. Other than that I have just removed some unused code from before. Also ensured I am using enif_make_binnary wherever needed instead of enif_make_string:

  • Extending the error coverage: The C function error_to_string handles that now and an example of a faulty graph reading attempt would look like this:
iex(1)>graph=Tensorflex.read_graph"Makefile"{:error,:invalid_argument}
  • Returning list of all operations in graph instead of directly printing them: This has also been taken care of, and now the function is called get_graph_ops which returns a List of strings (op names in the graph). An example is as follows:
iex(1)>graph=Tensorflex.read_graph("classify_image_graph_def.pb")2018-05-1723:36:16.488469: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supportsinstructionsthatthisTensorFlowbinarywasnotcompiledtouse: SSE4.1 SSE4.2 AVX AVX2 FMA 2018-05-1723:36:16.774442: W tensorflow/core/framework/op_def_util.cc:334] OpBatchNormWithGlobalNormalization isdeprecated.It will cease to work inGraphDefversion9.Use tf.nn.batch_normalization().Successfullyimportedgraph#Reference<0.1610607974.1988231169.250293>iex(2)>op_list=Tensorflex.get_graph_opsgraph["softmax/biases","softmax/weights","pool_3/_reshape/shape","mixed_10/join/concat_dim","mixed_10/tower_2/conv/batchnorm/moving_variance","mixed_10/tower_2/conv/batchnorm/moving_mean","mixed_10/tower_2/conv/batchnorm/gamma","mixed_10/tower_2/conv/batchnorm/beta","mixed_10/tower_2/conv/conv2d_params","mixed_10/tower_1/mixed/conv_1/batchnorm/moving_variance","mixed_10/tower_1/mixed/conv_1/batchnorm/moving_mean","mixed_10/tower_1/mixed/conv_1/batchnorm/gamma","mixed_10/tower_1/mixed/conv_1/batchnorm/beta","mixed_10/tower_1/mixed/conv_1/conv2d_params","mixed_10/tower_1/mixed/conv/batchnorm/moving_variance","mixed_10/tower_1/mixed/conv/batchnorm/moving_mean","mixed_10/tower_1/mixed/conv/batchnorm/gamma","mixed_10/tower_1/mixed/conv/batchnorm/beta","mixed_10/tower_1/mixed/conv/conv2d_params","mixed_10/tower_1/conv_1/batchnorm/moving_variance","mixed_10/tower_1/conv_1/batchnorm/moving_mean","mixed_10/tower_1/conv_1/batchnorm/gamma","mixed_10/tower_1/conv_1/batchnorm/beta","mixed_10/tower_1/conv_1/conv2d_params","mixed_10/tower_1/conv/batchnorm/moving_variance","mixed_10/tower_1/conv/batchnorm/moving_mean","mixed_10/tower_1/conv/batchnorm/gamma","mixed_10/tower_1/conv/batchnorm/beta","mixed_10/tower_1/conv/conv2d_params","mixed_10/tower/mixed/conv_1/batchnorm/moving_variance","mixed_10/tower/mixed/conv_1/batchnorm/moving_mean","mixed_10/tower/mixed/conv_1/batchnorm/gamma","mixed_10/tower/mixed/conv_1/batchnorm/beta","mixed_10/tower/mixed/conv_1/conv2d_params","mixed_10/tower/mixed/conv/batchnorm/moving_variance","mixed_10/tower/mixed/conv/batchnorm/moving_mean","mixed_10/tower/mixed/conv/batchnorm/gamma","mixed_10/tower/mixed/conv/batchnorm/beta","mixed_10/tower/mixed/conv/conv2d_params","mixed_10/tower/conv/batchnorm/moving_variance","mixed_10/tower/conv/batchnorm/moving_mean","mixed_10/tower/conv/batchnorm/gamma","mixed_10/tower/conv/batchnorm/beta","mixed_10/tower/conv/conv2d_params","mixed_10/conv/batchnorm/moving_variance","mixed_10/conv/batchnorm/moving_mean","mixed_10/conv/batchnorm/gamma","mixed_10/conv/batchnorm/beta","mixed_10/conv/conv2d_params","mixed_9/join/concat_dim",...]

Comment threadc_src/Tensorflex.c Outdated
break;
case TF_DATA_LOSS: strcpy(error,"data_loss");
break;
default: strcpy(error,"unlisted_code");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we could have each of those call enif_make_atom so we don't have to allocate the string in the first place, just to convert it to an atom?

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed this in the latest commit. I was needlessly allocating a string for this. Renamed the function to error_to_atom now.

Comment threadc_src/Tensorflex.c Outdated
char op_name[BASE_STRING_LENGTH];
for(int i=0; i<n_ops; i++) {
op_temp = TF_GraphNextOperation(*graph, &pos);
strcpy(op_name, (char*) TF_OperationName(op_temp));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to copy the string here? Given that we are already copying it later to the binary with memcpy?

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, this wasn't needed and I have fixed this in the latest commit. I apologize for the poorly written code here.

@josevalim

Copy link
Copy Markdown
Contributor

Comments added. Other than that, it looks great to go! 👍

What next steps do you have in mind? :)

@anshuman23

Copy link
Copy Markdown
OwnerAuthor

Thanks for the feedback @josevalim!

I have been thinking about the next steps, and the logical progression of adding functions should eventually get to the point where we can run the loaded graph against our own inputs and generate prediction outputs. For this to work, a lot of functions need to be added first: particularly ones that will allow us to read Tensors supplied by the user. I am thinking over how I would accomplish this and then I will submit a PR for review. These TF_Tensor functions are very important and they would have to encompass a wide variety of inputs.

I might also keep importing graph based functions from Tensorflow as and when required (like the operation list function get_graph_ops) After that, I think some more graph based functions will be needed that will use both the tensor functionality and the graph functions. Finally, we will need to add support for creating and running Sessions.

@anshuman23
anshuman23 merged commit 2398996 into masterMay 19, 2018
@josevalim

Copy link
Copy Markdown
Contributor

@anshuman23 agreed! ❤️

Let's go with this plan and continue exploring the API and the "user story" but note that at some point we will have to "sit down" and write documentation and tests.

Also, can you please send an e-mail to the beamcommunity mailing list with your progress so far and your plans for the next week? What is coming next can be a very quick digest, like this one that you just sent!

@anshuman23

Copy link
Copy Markdown
OwnerAuthor

Thanks @josevalim, and I completely agree on taking time out to write documentation and tests. I will incorporate that into my plans as well. :)

Yes, I'll send that email right away. Thanks again for all the help!

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@anshuman23@josevalim
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Returning list of op names in get_graph_ops and extended error atoms to all TF error codes - #6

Merged
anshuman23 merged 3 commits into
masterfrom
dev
May 19, 2018
Merged

Returning list of op names in get_graph_ops and extended error atoms to all TF error codes#6
anshuman23 merged 3 commits into
masterfrom
dev

Conversation

@anshuman23

Copy link
Copy Markdown
Owner

So this PR covers the changes you had last requested @josevalim. Other than that I have just removed some unused code from before. Also ensured I am using enif_make_binnary wherever needed instead of enif_make_string:

  • Extending the error coverage: The C function error_to_string handles that now and an example of a faulty graph reading attempt would look like this:
iex(1)>graph=Tensorflex.read_graph"Makefile"{:error,:invalid_argument}
  • Returning list of all operations in graph instead of directly printing them: This has also been taken care of, and now the function is called get_graph_ops which returns a List of strings (op names in the graph). An example is as follows:
iex(1)>graph=Tensorflex.read_graph("classify_image_graph_def.pb")2018-05-1723:36:16.488469: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supportsinstructionsthatthisTensorFlowbinarywasnotcompiledtouse: SSE4.1 SSE4.2 AVX AVX2 FMA 2018-05-1723:36:16.774442: W tensorflow/core/framework/op_def_util.cc:334] OpBatchNormWithGlobalNormalization isdeprecated.It will cease to work inGraphDefversion9.Use tf.nn.batch_normalization().Successfullyimportedgraph#Reference<0.1610607974.1988231169.250293>iex(2)>op_list=Tensorflex.get_graph_opsgraph["softmax/biases","softmax/weights","pool_3/_reshape/shape","mixed_10/join/concat_dim","mixed_10/tower_2/conv/batchnorm/moving_variance","mixed_10/tower_2/conv/batchnorm/moving_mean","mixed_10/tower_2/conv/batchnorm/gamma","mixed_10/tower_2/conv/batchnorm/beta","mixed_10/tower_2/conv/conv2d_params","mixed_10/tower_1/mixed/conv_1/batchnorm/moving_variance","mixed_10/tower_1/mixed/conv_1/batchnorm/moving_mean","mixed_10/tower_1/mixed/conv_1/batchnorm/gamma","mixed_10/tower_1/mixed/conv_1/batchnorm/beta","mixed_10/tower_1/mixed/conv_1/conv2d_params","mixed_10/tower_1/mixed/conv/batchnorm/moving_variance","mixed_10/tower_1/mixed/conv/batchnorm/moving_mean","mixed_10/tower_1/mixed/conv/batchnorm/gamma","mixed_10/tower_1/mixed/conv/batchnorm/beta","mixed_10/tower_1/mixed/conv/conv2d_params","mixed_10/tower_1/conv_1/batchnorm/moving_variance","mixed_10/tower_1/conv_1/batchnorm/moving_mean","mixed_10/tower_1/conv_1/batchnorm/gamma","mixed_10/tower_1/conv_1/batchnorm/beta","mixed_10/tower_1/conv_1/conv2d_params","mixed_10/tower_1/conv/batchnorm/moving_variance","mixed_10/tower_1/conv/batchnorm/moving_mean","mixed_10/tower_1/conv/batchnorm/gamma","mixed_10/tower_1/conv/batchnorm/beta","mixed_10/tower_1/conv/conv2d_params","mixed_10/tower/mixed/conv_1/batchnorm/moving_variance","mixed_10/tower/mixed/conv_1/batchnorm/moving_mean","mixed_10/tower/mixed/conv_1/batchnorm/gamma","mixed_10/tower/mixed/conv_1/batchnorm/beta","mixed_10/tower/mixed/conv_1/conv2d_params","mixed_10/tower/mixed/conv/batchnorm/moving_variance","mixed_10/tower/mixed/conv/batchnorm/moving_mean","mixed_10/tower/mixed/conv/batchnorm/gamma","mixed_10/tower/mixed/conv/batchnorm/beta","mixed_10/tower/mixed/conv/conv2d_params","mixed_10/tower/conv/batchnorm/moving_variance","mixed_10/tower/conv/batchnorm/moving_mean","mixed_10/tower/conv/batchnorm/gamma","mixed_10/tower/conv/batchnorm/beta","mixed_10/tower/conv/conv2d_params","mixed_10/conv/batchnorm/moving_variance","mixed_10/conv/batchnorm/moving_mean","mixed_10/conv/batchnorm/gamma","mixed_10/conv/batchnorm/beta","mixed_10/conv/conv2d_params","mixed_9/join/concat_dim",...]

Comment threadc_src/Tensorflex.c Outdated
break;
case TF_DATA_LOSS: strcpy(error,"data_loss");
break;
default: strcpy(error,"unlisted_code");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we could have each of those call enif_make_atom so we don't have to allocate the string in the first place, just to convert it to an atom?

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed this in the latest commit. I was needlessly allocating a string for this. Renamed the function to error_to_atom now.

Comment threadc_src/Tensorflex.c Outdated
char op_name[BASE_STRING_LENGTH];
for(int i=0; i<n_ops; i++) {
op_temp = TF_GraphNextOperation(*graph, &pos);
strcpy(op_name, (char*) TF_OperationName(op_temp));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to copy the string here? Given that we are already copying it later to the binary with memcpy?

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, this wasn't needed and I have fixed this in the latest commit. I apologize for the poorly written code here.

@josevalim

Copy link
Copy Markdown
Contributor

Comments added. Other than that, it looks great to go! 👍

What next steps do you have in mind? :)

@anshuman23

Copy link
Copy Markdown
OwnerAuthor

Thanks for the feedback @josevalim!

I have been thinking about the next steps, and the logical progression of adding functions should eventually get to the point where we can run the loaded graph against our own inputs and generate prediction outputs. For this to work, a lot of functions need to be added first: particularly ones that will allow us to read Tensors supplied by the user. I am thinking over how I would accomplish this and then I will submit a PR for review. These TF_Tensor functions are very important and they would have to encompass a wide variety of inputs.

I might also keep importing graph based functions from Tensorflow as and when required (like the operation list function get_graph_ops) After that, I think some more graph based functions will be needed that will use both the tensor functionality and the graph functions. Finally, we will need to add support for creating and running Sessions.

@anshuman23
anshuman23 merged commit 2398996 into masterMay 19, 2018
@josevalim

Copy link
Copy Markdown
Contributor

@anshuman23 agreed! ❤️

Let's go with this plan and continue exploring the API and the "user story" but note that at some point we will have to "sit down" and write documentation and tests.

Also, can you please send an e-mail to the beamcommunity mailing list with your progress so far and your plans for the next week? What is coming next can be a very quick digest, like this one that you just sent!

@anshuman23

Copy link
Copy Markdown
OwnerAuthor

Thanks @josevalim, and I completely agree on taking time out to write documentation and tests. I will incorporate that into my plans as well. :)

Yes, I'll send that email right away. Thanks again for all the help!

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@anshuman23@josevalim