[WIP] Force add a file that is in an ignored subdirectory

[jdblischak]

Motivation

A common setup I use for my Git repositories is to ignore entire directories which contain many files that I do not want to commit, e.g. subdirectories with data or figure files. Ignoring these directories prevents me from accidentally committing them and makes the output of git status less cluttered. However, I often want to commit a few files from these ignored subdirectories, e.g. a small data file with summary statistics or the final version of a figure I want to share. This is possible with Git, e.g.

$ cat .gitignore
data/
$ git add -f data/summary.txt

Problem

I am unable to force add a file in an ignored subdirectory with git2r::add(). Using the example above, git2r::add("data/summary.txt", force = TRUE) does not force add the file, nor does it throw a warning or error. It behaves the same as if force = FALSE.

What's currently in this PR

I haven't been able to figure out how to implement this myself, but to assist in developing a solution, I've added unit tests that currently fail but should pass if the behavior is implemented.

Researching potential solutions

PR #148 was the original request for force adding files, and commit 7ccb202 implemented the functionality. However, the focus was on ignored files, and not ignored subdirectories.

I have been trying to determine if this is a fundamental limitation of libgit2. The libgit2 docs for git_index_add_all discuss force adding specific files with the GIT_INDEX_ADD_FORCE flag, but do not discuss this specific use case. Looking at the Python API for libgit2, pygit2, it appears that they do not support force adding at all (source code, search results of online documentation), so that's not informative. The Lua API, luagit2, also appears to not handle ignored files, instead recommending that this be implemented manually: "This forces the file to be added to the index, not looking at gitignore rules. Those rules can be evaluated through the git_status APIs (in status.h) before calling this."

This issue is potentially related: libgit2/libgit2#3535 (comment) At the very least it demonstrates an example where Git and libgit2 differ in their interpretation of the .gitingore file.

Would it be possible to implement this functionality in git2r either by modifying the call to libgit2 or instead adding a custom check via a call to git2r::status()?

[stewid]

Sorry for the late reply.
Thanks for finding this and creating the test case. I think this is a missing corner case in libgit2. Maybe it works in the next version of libgit2.

[jdblischak]

No worries @stewid! I did a lot of searching, so I figured there wasn't any easy solution.

Conceptually to me at least it is similar to the all_untracked option added in git2r 0.16.0. Instead of status reporting an ignored subdirectory, it needs to recognize that there are individual ignored files in that subdirectory.

[stewid]

That's a good point, maybe it's possible to check status of untracked files before and after .Call(git2r_index_add_all, repo, path, isTRUE(force)) to determine if there are untracked files that should be added. I'm working on another problem in the add method so maybe if I can fix this at the same time.

[jdblischak]

Here's a related Issue in the upstream libgit2 tracker: libgit2/libgit2#4377

It appears to be documenting the same behavior, i.e. that libgit2 can't force add a file inside of an ignored subdirectory.

[jdblischak]

Motivation

A common setup I use for my Git repositories is to ignore entire directories which contain many files that I do not want to commit, e.g. subdirectories with data or figure files. Ignoring these directories prevents me from accidentally committing them and makes the output of git status less cluttered. However, I often want to commit a few files from these ignored subdirectories, e.g. a small data file with summary statistics or the final version of a figure I want to share. This is possible with Git, e.g.

$ cat .gitignore
data/
$ git add -f data/summary.txt

Problem

I am unable to force add a file in an ignored subdirectory with git2r::add(). Using the example above, git2r::add("data/summary.txt", force = TRUE) does not force add the file, nor does it throw a warning or error. It behaves the same as if force = FALSE.

What's currently in this PR

I haven't been able to figure out how to implement this myself, but to assist in developing a solution, I've added unit tests that currently fail but should pass if the behavior is implemented.

Researching potential solutions

PR #148 was the original request for force adding files, and commit 7ccb202 implemented the functionality. However, the focus was on ignored files, and not ignored subdirectories.

I have been trying to determine if this is a fundamental limitation of libgit2. The libgit2 docs for git_index_add_all discuss force adding specific files with the GIT_INDEX_ADD_FORCE flag, but do not discuss this specific use case. Looking at the Python API for libgit2, pygit2, it appears that they do not support force adding at all (source code, search results of online documentation), so that's not informative. The Lua API, luagit2, also appears to not handle ignored files, instead recommending that this be implemented manually: "This forces the file to be added to the index, not looking at gitignore rules. Those rules can be evaluated through the git_status APIs (in status.h) before calling this."

This issue is potentially related: libgit2/libgit2#3535 (comment) At the very least it demonstrates an example where Git and libgit2 differ in their interpretation of the .gitingore file.

Would it be possible to implement this functionality in git2r either by modifying the call to libgit2 or instead adding a custom check via a call to git2r::status()?

[stewid]

Sorry for the late reply.
Thanks for finding this and creating the test case. I think this is a missing corner case in libgit2. Maybe it works in the next version of libgit2.

[jdblischak]

No worries @stewid! I did a lot of searching, so I figured there wasn't any easy solution.

Conceptually to me at least it is similar to the all_untracked option added in git2r 0.16.0. Instead of status reporting an ignored subdirectory, it needs to recognize that there are individual ignored files in that subdirectory.

[stewid]

That's a good point, maybe it's possible to check status of untracked files before and after .Call(git2r_index_add_all, repo, path, isTRUE(force)) to determine if there are untracked files that should be added. I'm working on another problem in the add method so maybe if I can fix this at the same time.

[jdblischak]

Here's a related Issue in the upstream libgit2 tracker: libgit2/libgit2#4377

It appears to be documenting the same behavior, i.e. that libgit2 can't force add a file inside of an ignored subdirectory.

[jdblischak]

Motivation

A common setup I use for my Git repositories is to ignore entire directories which contain many files that I do not want to commit, e.g. subdirectories with data or figure files. Ignoring these directories prevents me from accidentally committing them and makes the output of git status less cluttered. However, I often want to commit a few files from these ignored subdirectories, e.g. a small data file with summary statistics or the final version of a figure I want to share. This is possible with Git, e.g.

$ cat .gitignore
data/
$ git add -f data/summary.txt

Problem

I am unable to force add a file in an ignored subdirectory with git2r::add(). Using the example above, git2r::add("data/summary.txt", force = TRUE) does not force add the file, nor does it throw a warning or error. It behaves the same as if force = FALSE.

What's currently in this PR

I haven't been able to figure out how to implement this myself, but to assist in developing a solution, I've added unit tests that currently fail but should pass if the behavior is implemented.

Researching potential solutions

PR #148 was the original request for force adding files, and commit 7ccb202 implemented the functionality. However, the focus was on ignored files, and not ignored subdirectories.

I have been trying to determine if this is a fundamental limitation of libgit2. The libgit2 docs for git_index_add_all discuss force adding specific files with the GIT_INDEX_ADD_FORCE flag, but do not discuss this specific use case. Looking at the Python API for libgit2, pygit2, it appears that they do not support force adding at all (source code, search results of online documentation), so that's not informative. The Lua API, luagit2, also appears to not handle ignored files, instead recommending that this be implemented manually: "This forces the file to be added to the index, not looking at gitignore rules. Those rules can be evaluated through the git_status APIs (in status.h) before calling this."

This issue is potentially related: libgit2/libgit2#3535 (comment) At the very least it demonstrates an example where Git and libgit2 differ in their interpretation of the .gitingore file.

Would it be possible to implement this functionality in git2r either by modifying the call to libgit2 or instead adding a custom check via a call to git2r::status()?

[stewid]

Sorry for the late reply.
Thanks for finding this and creating the test case. I think this is a missing corner case in libgit2. Maybe it works in the next version of libgit2.

[jdblischak]

No worries @stewid! I did a lot of searching, so I figured there wasn't any easy solution.

Conceptually to me at least it is similar to the all_untracked option added in git2r 0.16.0. Instead of status reporting an ignored subdirectory, it needs to recognize that there are individual ignored files in that subdirectory.

[stewid]

That's a good point, maybe it's possible to check status of untracked files before and after .Call(git2r_index_add_all, repo, path, isTRUE(force)) to determine if there are untracked files that should be added. I'm working on another problem in the add method so maybe if I can fix this at the same time.

[jdblischak]

Here's a related Issue in the upstream libgit2 tracker: libgit2/libgit2#4377

It appears to be documenting the same behavior, i.e. that libgit2 can't force add a file inside of an ignored subdirectory.

[jdblischak]

Motivation

A common setup I use for my Git repositories is to ignore entire directories which contain many files that I do not want to commit, e.g. subdirectories with data or figure files. Ignoring these directories prevents me from accidentally committing them and makes the output of git status less cluttered. However, I often want to commit a few files from these ignored subdirectories, e.g. a small data file with summary statistics or the final version of a figure I want to share. This is possible with Git, e.g.

$ cat .gitignore
data/
$ git add -f data/summary.txt

Problem

I am unable to force add a file in an ignored subdirectory with git2r::add(). Using the example above, git2r::add("data/summary.txt", force = TRUE) does not force add the file, nor does it throw a warning or error. It behaves the same as if force = FALSE.

What's currently in this PR

I haven't been able to figure out how to implement this myself, but to assist in developing a solution, I've added unit tests that currently fail but should pass if the behavior is implemented.

Researching potential solutions

PR #148 was the original request for force adding files, and commit 7ccb202 implemented the functionality. However, the focus was on ignored files, and not ignored subdirectories.

I have been trying to determine if this is a fundamental limitation of libgit2. The libgit2 docs for git_index_add_all discuss force adding specific files with the GIT_INDEX_ADD_FORCE flag, but do not discuss this specific use case. Looking at the Python API for libgit2, pygit2, it appears that they do not support force adding at all (source code, search results of online documentation), so that's not informative. The Lua API, luagit2, also appears to not handle ignored files, instead recommending that this be implemented manually: "This forces the file to be added to the index, not looking at gitignore rules. Those rules can be evaluated through the git_status APIs (in status.h) before calling this."

This issue is potentially related: libgit2/libgit2#3535 (comment) At the very least it demonstrates an example where Git and libgit2 differ in their interpretation of the .gitingore file.

Would it be possible to implement this functionality in git2r either by modifying the call to libgit2 or instead adding a custom check via a call to git2r::status()?

[stewid]

Sorry for the late reply.
Thanks for finding this and creating the test case. I think this is a missing corner case in libgit2. Maybe it works in the next version of libgit2.

[jdblischak]

No worries @stewid! I did a lot of searching, so I figured there wasn't any easy solution.

Conceptually to me at least it is similar to the all_untracked option added in git2r 0.16.0. Instead of status reporting an ignored subdirectory, it needs to recognize that there are individual ignored files in that subdirectory.

[stewid]

That's a good point, maybe it's possible to check status of untracked files before and after .Call(git2r_index_add_all, repo, path, isTRUE(force)) to determine if there are untracked files that should be added. I'm working on another problem in the add method so maybe if I can fix this at the same time.

[jdblischak]

Here's a related Issue in the upstream libgit2 tracker: libgit2/libgit2#4377

It appears to be documenting the same behavior, i.e. that libgit2 can't force add a file inside of an ignored subdirectory.

[jdblischak]

Motivation

A common setup I use for my Git repositories is to ignore entire directories which contain many files that I do not want to commit, e.g. subdirectories with data or figure files. Ignoring these directories prevents me from accidentally committing them and makes the output of git status less cluttered. However, I often want to commit a few files from these ignored subdirectories, e.g. a small data file with summary statistics or the final version of a figure I want to share. This is possible with Git, e.g.

$ cat .gitignore
data/
$ git add -f data/summary.txt

Problem

I am unable to force add a file in an ignored subdirectory with git2r::add(). Using the example above, git2r::add("data/summary.txt", force = TRUE) does not force add the file, nor does it throw a warning or error. It behaves the same as if force = FALSE.

What's currently in this PR

I haven't been able to figure out how to implement this myself, but to assist in developing a solution, I've added unit tests that currently fail but should pass if the behavior is implemented.

Researching potential solutions

PR #148 was the original request for force adding files, and commit 7ccb202 implemented the functionality. However, the focus was on ignored files, and not ignored subdirectories.

I have been trying to determine if this is a fundamental limitation of libgit2. The libgit2 docs for git_index_add_all discuss force adding specific files with the GIT_INDEX_ADD_FORCE flag, but do not discuss this specific use case. Looking at the Python API for libgit2, pygit2, it appears that they do not support force adding at all (source code, search results of online documentation), so that's not informative. The Lua API, luagit2, also appears to not handle ignored files, instead recommending that this be implemented manually: "This forces the file to be added to the index, not looking at gitignore rules. Those rules can be evaluated through the git_status APIs (in status.h) before calling this."

This issue is potentially related: libgit2/libgit2#3535 (comment) At the very least it demonstrates an example where Git and libgit2 differ in their interpretation of the .gitingore file.

Would it be possible to implement this functionality in git2r either by modifying the call to libgit2 or instead adding a custom check via a call to git2r::status()?

[stewid]

Sorry for the late reply.
Thanks for finding this and creating the test case. I think this is a missing corner case in libgit2. Maybe it works in the next version of libgit2.

[jdblischak]

No worries @stewid! I did a lot of searching, so I figured there wasn't any easy solution.

Conceptually to me at least it is similar to the all_untracked option added in git2r 0.16.0. Instead of status reporting an ignored subdirectory, it needs to recognize that there are individual ignored files in that subdirectory.

[stewid]

That's a good point, maybe it's possible to check status of untracked files before and after .Call(git2r_index_add_all, repo, path, isTRUE(force)) to determine if there are untracked files that should be added. I'm working on another problem in the add method so maybe if I can fix this at the same time.

[jdblischak]

Here's a related Issue in the upstream libgit2 tracker: libgit2/libgit2#4377

It appears to be documenting the same behavior, i.e. that libgit2 can't force add a file inside of an ignored subdirectory.

[jdblischak]

Motivation

A common setup I use for my Git repositories is to ignore entire directories which contain many files that I do not want to commit, e.g. subdirectories with data or figure files. Ignoring these directories prevents me from accidentally committing them and makes the output of git status less cluttered. However, I often want to commit a few files from these ignored subdirectories, e.g. a small data file with summary statistics or the final version of a figure I want to share. This is possible with Git, e.g.

$ cat .gitignore
data/
$ git add -f data/summary.txt

Problem

I am unable to force add a file in an ignored subdirectory with git2r::add(). Using the example above, git2r::add("data/summary.txt", force = TRUE) does not force add the file, nor does it throw a warning or error. It behaves the same as if force = FALSE.

What's currently in this PR

I haven't been able to figure out how to implement this myself, but to assist in developing a solution, I've added unit tests that currently fail but should pass if the behavior is implemented.

Researching potential solutions

PR #148 was the original request for force adding files, and commit 7ccb202 implemented the functionality. However, the focus was on ignored files, and not ignored subdirectories.

I have been trying to determine if this is a fundamental limitation of libgit2. The libgit2 docs for git_index_add_all discuss force adding specific files with the GIT_INDEX_ADD_FORCE flag, but do not discuss this specific use case. Looking at the Python API for libgit2, pygit2, it appears that they do not support force adding at all (source code, search results of online documentation), so that's not informative. The Lua API, luagit2, also appears to not handle ignored files, instead recommending that this be implemented manually: "This forces the file to be added to the index, not looking at gitignore rules. Those rules can be evaluated through the git_status APIs (in status.h) before calling this."

This issue is potentially related: libgit2/libgit2#3535 (comment) At the very least it demonstrates an example where Git and libgit2 differ in their interpretation of the .gitingore file.

Would it be possible to implement this functionality in git2r either by modifying the call to libgit2 or instead adding a custom check via a call to git2r::status()?

[stewid]

Sorry for the late reply.
Thanks for finding this and creating the test case. I think this is a missing corner case in libgit2. Maybe it works in the next version of libgit2.

[jdblischak]

No worries @stewid! I did a lot of searching, so I figured there wasn't any easy solution.

Conceptually to me at least it is similar to the all_untracked option added in git2r 0.16.0. Instead of status reporting an ignored subdirectory, it needs to recognize that there are individual ignored files in that subdirectory.

[stewid]

That's a good point, maybe it's possible to check status of untracked files before and after .Call(git2r_index_add_all, repo, path, isTRUE(force)) to determine if there are untracked files that should be added. I'm working on another problem in the add method so maybe if I can fix this at the same time.

[jdblischak]

Here's a related Issue in the upstream libgit2 tracker: libgit2/libgit2#4377

It appears to be documenting the same behavior, i.e. that libgit2 can't force add a file inside of an ignored subdirectory.

[jdblischak]

Motivation

A common setup I use for my Git repositories is to ignore entire directories which contain many files that I do not want to commit, e.g. subdirectories with data or figure files. Ignoring these directories prevents me from accidentally committing them and makes the output of git status less cluttered. However, I often want to commit a few files from these ignored subdirectories, e.g. a small data file with summary statistics or the final version of a figure I want to share. This is possible with Git, e.g.

$ cat .gitignore
data/
$ git add -f data/summary.txt

Problem

I am unable to force add a file in an ignored subdirectory with git2r::add(). Using the example above, git2r::add("data/summary.txt", force = TRUE) does not force add the file, nor does it throw a warning or error. It behaves the same as if force = FALSE.

What's currently in this PR

I haven't been able to figure out how to implement this myself, but to assist in developing a solution, I've added unit tests that currently fail but should pass if the behavior is implemented.

Researching potential solutions

PR #148 was the original request for force adding files, and commit 7ccb202 implemented the functionality. However, the focus was on ignored files, and not ignored subdirectories.

I have been trying to determine if this is a fundamental limitation of libgit2. The libgit2 docs for git_index_add_all discuss force adding specific files with the GIT_INDEX_ADD_FORCE flag, but do not discuss this specific use case. Looking at the Python API for libgit2, pygit2, it appears that they do not support force adding at all (source code, search results of online documentation), so that's not informative. The Lua API, luagit2, also appears to not handle ignored files, instead recommending that this be implemented manually: "This forces the file to be added to the index, not looking at gitignore rules. Those rules can be evaluated through the git_status APIs (in status.h) before calling this."

This issue is potentially related: libgit2/libgit2#3535 (comment) At the very least it demonstrates an example where Git and libgit2 differ in their interpretation of the .gitingore file.

Would it be possible to implement this functionality in git2r either by modifying the call to libgit2 or instead adding a custom check via a call to git2r::status()?

[stewid]

Sorry for the late reply.
Thanks for finding this and creating the test case. I think this is a missing corner case in libgit2. Maybe it works in the next version of libgit2.

[jdblischak]

No worries @stewid! I did a lot of searching, so I figured there wasn't any easy solution.

Conceptually to me at least it is similar to the all_untracked option added in git2r 0.16.0. Instead of status reporting an ignored subdirectory, it needs to recognize that there are individual ignored files in that subdirectory.

[stewid]

That's a good point, maybe it's possible to check status of untracked files before and after .Call(git2r_index_add_all, repo, path, isTRUE(force)) to determine if there are untracked files that should be added. I'm working on another problem in the add method so maybe if I can fix this at the same time.

[jdblischak]

Here's a related Issue in the upstream libgit2 tracker: libgit2/libgit2#4377

It appears to be documenting the same behavior, i.e. that libgit2 can't force add a file inside of an ignored subdirectory.

[jdblischak]

Motivation

A common setup I use for my Git repositories is to ignore entire directories which contain many files that I do not want to commit, e.g. subdirectories with data or figure files. Ignoring these directories prevents me from accidentally committing them and makes the output of git status less cluttered. However, I often want to commit a few files from these ignored subdirectories, e.g. a small data file with summary statistics or the final version of a figure I want to share. This is possible with Git, e.g.

$ cat .gitignore
data/
$ git add -f data/summary.txt

Problem

I am unable to force add a file in an ignored subdirectory with git2r::add(). Using the example above, git2r::add("data/summary.txt", force = TRUE) does not force add the file, nor does it throw a warning or error. It behaves the same as if force = FALSE.

What's currently in this PR

I haven't been able to figure out how to implement this myself, but to assist in developing a solution, I've added unit tests that currently fail but should pass if the behavior is implemented.

Researching potential solutions

PR #148 was the original request for force adding files, and commit 7ccb202 implemented the functionality. However, the focus was on ignored files, and not ignored subdirectories.

I have been trying to determine if this is a fundamental limitation of libgit2. The libgit2 docs for git_index_add_all discuss force adding specific files with the GIT_INDEX_ADD_FORCE flag, but do not discuss this specific use case. Looking at the Python API for libgit2, pygit2, it appears that they do not support force adding at all (source code, search results of online documentation), so that's not informative. The Lua API, luagit2, also appears to not handle ignored files, instead recommending that this be implemented manually: "This forces the file to be added to the index, not looking at gitignore rules. Those rules can be evaluated through the git_status APIs (in status.h) before calling this."

This issue is potentially related: libgit2/libgit2#3535 (comment) At the very least it demonstrates an example where Git and libgit2 differ in their interpretation of the .gitingore file.

Would it be possible to implement this functionality in git2r either by modifying the call to libgit2 or instead adding a custom check via a call to git2r::status()?

[stewid]

Sorry for the late reply.
Thanks for finding this and creating the test case. I think this is a missing corner case in libgit2. Maybe it works in the next version of libgit2.

[jdblischak]

No worries @stewid! I did a lot of searching, so I figured there wasn't any easy solution.

Conceptually to me at least it is similar to the all_untracked option added in git2r 0.16.0. Instead of status reporting an ignored subdirectory, it needs to recognize that there are individual ignored files in that subdirectory.

[stewid]

That's a good point, maybe it's possible to check status of untracked files before and after .Call(git2r_index_add_all, repo, path, isTRUE(force)) to determine if there are untracked files that should be added. I'm working on another problem in the add method so maybe if I can fix this at the same time.

[jdblischak]

Here's a related Issue in the upstream libgit2 tracker: libgit2/libgit2#4377

It appears to be documenting the same behavior, i.e. that libgit2 can't force add a file inside of an ignored subdirectory.