Skip to content

Full text crawlers - #101

Merged
stefan-kolb merged 12 commits into
masterfrom
catalog-crawler
Oct 9, 2015
Merged

Full text crawlers#101
stefan-kolb merged 12 commits into
masterfrom
catalog-crawler

Conversation

@stefan-kolb

@stefan-kolbstefan-kolb commented Aug 14, 2015

Copy link
Copy Markdown
Member

This PR enables automatic PDF fulltext downloads.

Current catalogs:

  • ACS
  • arXiv
  • Springer
  • Sciencedirect (Elsevier)
  • Google Scholar

Questions:

  • Do we want to include more Crawlers? If yes, which?
  • Action was in the Tools menu. Imho it should be either auto downloaded or included with the downloador auto button in the detailed entry view.

TODO:

  • Make it nonblocking and keep the progress bar!
  • Add an option to delete the local file if a wrong file was downloaded
  • Tests on Travis & CircleCI are blocked by Google by 403 Forbidden (Bot detection)

@stefan-kolbstefan-kolb changed the title Science catalog full text crawlers[WIP] Science catalog full text crawlersAug 14, 2015
@stefan-kolbstefan-kolb changed the title [WIP] Science catalog full text crawlers[WIP] Full text crawlersAug 14, 2015
@mlep

mlep commented Aug 17, 2015

Copy link
Copy Markdown
Contributor

About "Do we want to include more Crawlers? If yes, which?":
You have a gold mine in the section "Search (and import) from my specific database, please." in the sorted list of feature requests (this morning email).

@stefan-kolb

stefan-kolb commented Aug 17, 2015

Copy link
Copy Markdown
MemberAuthor

@mlep Thanks 👍

@simonharrer

simonharrer commented Aug 18, 2015

Copy link
Copy Markdown
Contributor

Your feature works pretty well. But there are cases in which a file can be downloaded but it is not the file that is intended. Thus, there must be a way to preview the pdf before adding it.

@stefan-kolb

stefan-kolb commented Aug 18, 2015

Copy link
Copy Markdown
MemberAuthor

We need to evaluate how often this will happen. Can you give me an example?

I think with a reasonable amount of crawlers this will work with a very high accuracy.
I would rather have the user open and check the file and then delete it and manually download it if it is not the correct file, as this will happen very seldom. If we use your approach we need to preview every PDF that will be anyway in > 90% of the times the correct one.

@simonharrer

simonharrer commented Aug 18, 2015

Copy link
Copy Markdown
Contributor

Ok, deletion after download is OK when it is done easily. At the moment, I cannot easily delete the file entry from the bibtex entry AND also delete the file from the hard drive. When we fix this, we do not need the check feature in the first place. Good idea!

My example was: I wanted to download the conference article Barros, Service Interaction Patterns, but got the techrep instead. Another issue was that I got a PDF of slides in some language I do not understand, probably czech when attempting to download a very old version of the BPMN standard.

@stefan-kolbstefan-kolb mentioned this pull request Aug 18, 2015
Closed
@stefan-kolb
stefan-kolbforce-pushed the catalog-crawler branch 2 times, most recently from 43d0c7d to 3384c4cCompareAugust 18, 2015 12:49
@koppor

koppor commented Aug 18, 2015

Copy link
Copy Markdown
Member

Regarding, "clean all used fields from Latex stuff, e.g. {~ etc.", this should be one of the "Edit -> Cleanup entries" functionality, isn't it? If not, this functionality should be added there, too.

Regarding "check for duplicates", JabRef offers "Search -> find duplicates". This functionality should be reused here.

@koppor

koppor commented Aug 18, 2015

Copy link
Copy Markdown
Member

A similar approach might have been taken by Christoph Lehner: https://sourceforge.net/p/jabref/discussion/318824/thread/6e5fea64/ Is it possible to synchronize somehow?
I think, this makes https://github.com/wbrenna/LocalCopy obsolete, doesn't it?

@stefan-kolb

stefan-kolb commented Aug 19, 2015

Copy link
Copy Markdown
MemberAuthor

The approach of C.Lehner is the base repository https://github.com/lehner/LocalCopy.
This PR is a native replacement for this and all forks.
Also, we don't support plugins anymore.

@stefan-kolb
stefan-kolbforce-pushed the catalog-crawler branch 5 times, most recently from 4399958 to 5b6c7a4CompareAugust 24, 2015 15:51
@stefan-kolbstefan-kolb added this to the v2.80 milestone Aug 27, 2015
@minkkinenminkkinen mentioned this pull request Sep 18, 2015
Closed
@stefan-kolbstefan-kolb mentioned this pull request Sep 28, 2015
Merged
This was referenced Oct 7, 2015
Closed
Open
@lenhardlenhard mentioned this pull request Oct 9, 2015
Closed
@lenhard

lenhard commented Oct 9, 2015

Copy link
Copy Markdown
Member

What is missing to get this PR functional and integrated into master?

@lenhardlenhard assigned lenhard and simonharrer and unassigned lenhardOct 9, 2015
@simonharrer

simonharrer commented Oct 9, 2015

Copy link
Copy Markdown
Contributor

This is just too messy to understand right now. Fixing this would require fixing the separation of GUI Event Thread with other Code parts - a major effort.

The issue is how to implement swing actions that require multiple user interactions during their task taking place. Normally, this would require a nesting of swing worker classes, one for each step and starting the next step within the EDT update method.

What is more, sometimes SwingWorker, and sometimes the spin Framework is used. All of this makes this issue even more complicated.

Someone else may take a look if they see this issue more clearly than me.

@simonharrersimonharrer removed their assignment Oct 9, 2015
@lenhard

lenhard commented Oct 9, 2015

Copy link
Copy Markdown
Member

So you are saying that you will not complete the PR? I cannot see anybody else who will. There is no point of having this hanging around in limbo until it deprecates. So we can close this PR without merging and close related open issues as won't fix.

@simonharrer: Please confirm!

@simonharrer

simonharrer commented Oct 9, 2015

Copy link
Copy Markdown
Contributor

Merge as good enough.

@stefan-kolbstefan-kolb changed the title [WIP] Full text crawlersFull text crawlersOct 9, 2015
stefan-kolb added a commit that referenced this pull request Oct 9, 2015
@stefan-kolb
Merge pull request#101from JabRef/catalog-crawler
18a6a0e
Full text crawlers
@stefan-kolb
stefan-kolb merged commit 18a6a0e into masterOct 9, 2015
@stefan-kolb
stefan-kolb deleted the catalog-crawler branch October 9, 2015 13:34
@kopporkoppor mentioned this pull request Oct 10, 2015
Closed
@lecc0r

lecc0r commented Dec 1, 2015

Copy link
Copy Markdown

Hi, I've just tested the newly implemented "Full text article download" function, however for several articles it downloaded a version from ResearchGate but I want to have the one from Elsevier. Dependent who you are working for, you have special access privileges on certain publisher sites and it would be great to specify in the preferences which crawler the download function should prefer. Perhaps even define a priority list.

@stefan-kolb

stefan-kolb commented Dec 1, 2015

Copy link
Copy Markdown
MemberAuthor

There is a hard-coded priority list right know which prefers the official publishers over google scholar for example. If you want to file a new feature request or enhancement please create a separate issue.

@koppor

koppor commented Dec 1, 2015

Copy link
Copy Markdown
Member

Done at #435.

InAnYan added a commit that referenced this pull request Aug 3, 2024
@InAnYan
Fix#99and partially#101
c245e73
github-merge-queueBot pushed a commit that referenced this pull request Aug 14, 2024
@InAnYan@koppor
AI chatting functionality (#11430)
8a0edc2
* Fix the code from code review
* Fix from code review and create new AiChatTabWorking
* Improve chat history storage code
* More fix from code review
* Remove obsolete parameter
* Add JavaDoc comment
* Fix checkstyle
* Fix JavaDoc
* Fix more checkstyle
* More checkstyle fixes
* Fix code changes
* Improve the PR
* Rework ADR-0031 to enable to use another option
* Add many LOGGEr.trace statements
* Change "message window" to "context window"
* Fix compiler errors
* Fix issue list index issue of langchain4j
* Fix lint issue
* Update 0031-store-chats-alongside-database.md
* More tracing
* Refine logging
* Remove closing of AiChatLanguageModel (because it's not closable)
* Use external package for OpenAI API connection
* Provide a custom executor for RetrievalAugmentor
* Fix shutdown issue (I hope)
* Refactor classes
* Change BibDatabaseChatHistoryFile
* Revert BibDatabaseChatHistoryFile to old version because of langchain4j
* Make round corners for chat messages
* Refactor embeddings generation
* Refactor embeddings generation
* Refactor embeddings generation
* Fix CHANGELOG.md
* Remove jpro-mdfx
* Add comment
* Fix localizations
* Fix checkstyle and remove OpenAI from PRIVACY.md
* Remove unnecessary comments
* Fix privacy notice UI
* Introduce new ApiKeyMissingComponent
* Thanks Tobiaz Diez for writing such a good EntryEditorTab class
* Fix InAnYan/jabref issues
* Merge `build.gradle` and `settings.gradle` from main branch
* Update ADRs
* Implement rethought ADR for chat history
* Use OpenAI embedding model
* Use Deep Java embedding model
* Remove old langchain4j embedding models
* Fix checkstyle errors
* Fix checkstyle and remove old dependencies
* Fixes from code review
* Restructure
* Fix checkstyle errors
* Add API base URL parameter
* Fix localization
* Fix from code review + ADR
* Something broken
* Now MistralAI and Hugging Face work
* Fix base URL for other LLM providers
* Fix base URL for other LLM providers
* Refactor MVStore usage
* Load embedding model in background
* Bump langchain4j version
* Fix bug
* Fix checkstyle and localization
* Implement summarization
* Fix checkstyle and localization
* Improve PrivacyNoticeComponent
* Fix from code review
* Update localization
* Wrap text
* Add padding
* Fix markdown
* Use stuff algorithm
* Add GPT-4o-mini
* Make chat model editable
* Update context window size and summarization
* Fix checkstyle
* Update PrivacyNoticeComponent.fxml
* Update AI summary tab
* Fix localization
* Change order so that there is no diff
* Reorrder dependencies
* Add missing CHANGELOG.md entry
* Refine ADR-0033
* Refine ADR0034
* Fix typos
* Refine ADR-0036
* Fix ADR-0037
* Fix title case
* Fix changes in module-info.java
* Readd removed requires org.apache.httpcomponents.core5.httpcore5
* Revert change in JabRefGUI to avoid conflicts
* Remove empty lines
* Reorder entries in JabRef_en.properties
* Simplify SummariesStorage (and add test)
* Use region/endregion
* Fix position of comment
* Add comment why the event bus is needed
* Do not show exception to the user - just that an error is occurred (saves %0 in localization)
* Use "URL %0" without colon (consistency)
* Fix typos
* History has to be kept
* Remove empty lines
* Fix language (hopefully)
* Compilefix
* Simplify BibDatabaseChatHistoryManager
* Fix from code review
* Fix issue #103
* Rework embeddings cache clearing
* Fix#99 and partially #101
* Partially fixing shutdown issues and UI progress monitor issue
* Add "requires scala.library" and add "region:" / "endregion"
* More grouping (move de.saxsys.mvvmfx.validation up)
* Add alphabetical hint
* FixInAnYan#101 and InAnYan#106
* Discard changes to settings.gradle
* FixInAnYan#105
* Follow-up fix for InAnYan#103
* Follow-up fix for InAnYan#103
* Remove obsolete class
* Partially fixInAnYan#98
* We do need dependencies to the AI providers, don't we?
* FixInAnYan#93
* Simplify code
* Partially fixInAnYan#92
* Fix checkstyle and localization
* Fix hyperlinks and text in ApiKeyMissingComponent
* Fixes from code review
* FixInAnYan#120
* Remove "X% work done" messages
* FixInAnYan#114
* Partially fixInAnYan#113
* Partially fixInAnYan#110
* FixInAnYan#110
* FixInAnYan#111
* Improve embedding model downloading notifications
* FixInAnYan#124
* FixInAnYan#122
* Fix wrong context window size when expert settings customization is turned off
* Attempt to fixInAnYan#95
* Finally fixInAnYan#105
* FixInAnYan#108
* Attempt to fixInAnYan#98
* Fix for InAnYan#104
* Fix for InAnYan#98
* Fix for InAnYan#95 (comment)
* Fix for InAnYan#98 (comment)
* Fix for InAnYan#126
* Fix for InAnYan#115
* Fix for InAnYan#113
* Fix for InAnYan#91
* Fix for InAnYan#121
* Fix for InAnYan#112 and InAnYan#116
* Fix for InAnYan#125
* Fixes from commit comments
* Fix for InAnYan#115
* Fix for InAnYan#120
* Fix for InAnYan#132
* Fix for InAnYan#132
* Fix for InAnYan#104
* Fix for InAnYan#118
* Fix for InAnYan#114
* Fix for InAnYan#104
* Store error messages in chat history
* Make error be a ChatMessageComponent
* Implement delete messages InAnYan#136
* Fix for InAnYan#118
* Fix for InAnYan#92
* Fix checkstyle and localization. And refactoring
* Fix for InAnYan#92
* Fix for InAnYan#139
* Show "Delete message" button only when necessary
* Fix for InAnYan#83
* Update src/main/java/org/jabref/logic/ai/AiService.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/logic/ai/chathistory/BibDatabaseChatHistoryManager.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/logic/ai/AiService.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/gui/Base.css
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/gui/Base.css
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Fix from code review
* Partial fix for InAnYan#125
* Update colors for error message
* Fix for InAnYan#145 and InAnYan#142
* Make progress for embedding model download
* Fix checkstyle and localization
* Add workaround to get FileHistoryMenuTest running again
* Small fixes
* Revert "Small fixes"
This reverts commit 85382a1.
* Introduce AiApiKeyProvider
* Fix IDE setup instructions
* Do not load API keys on startup
* Rely on keystore encryption
* Prevent mulitple rebuilds when muliple preferences are updated
* Fix localization to be more provider independent
* Fix method names
* Add poor man's solution to notify of API key changes
* Reduce calls to key store (and fix key saving)
* Fix for InAnYan#148 and partially InAnYan#146
* Revert "Fix for InAnYan#148 and partially InAnYan#146"
This reverts commit 5fa3bb5.
* Fix for scrolling down when deleting a message
* Sort EmbeddingModel enum variants
* Fix GenerateSummaryTask progress indication
* Fix dark mode
* Add notice for embedding models size
---------
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

[outdated] type: feature

Projects

None yet

Milestone

v3.0

Development

Successfully merging this pull request may close these issues.

6 participants

@stefan-kolb@mlep@simonharrer@koppor@lenhard@lecc0r
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Full text crawlers by stefan-kolb · Pull Request #101 · JabRef/jabref · GitHub
Skip to content
Full text crawlers - #101
Merged
stefan-kolb merged 12 commits into
masterfrom 
catalog-crawler
Oct 9, 2015
Merged
Full text crawlers#101
stefan-kolb merged 12 commits into
masterfrom 
catalog-crawler
Conversation
@stefan-kolb
@stefan-kolbstefan-kolb
commented
Aug 14, 2015

Copy link


Copy Markdown

Member
This PR enables automatic PDF fulltext downloads.
Current catalogs:
  • ACS
  • arXiv
  • Springer
  • Sciencedirect (Elsevier)
  • Google Scholar
Questions:
  • Do we want to include more Crawlers? If yes, which?
  • Action was in the Tools menu. Imho it should be either auto downloaded or included with the downloador auto button in the detailed entry view.
TODO:
  •  Make it nonblocking and keep the progress bar!
  •  Add an option to delete the local file if a wrong file was downloaded
  •  Tests on Travis & CircleCI are blocked by Google by 403 Forbidden (Bot detection)
@stefan-kolbstefan-kolb
changed the title
Science catalog full text crawlers[WIP] Science catalog full text crawlersAug 14, 2015
@stefan-kolbstefan-kolb
changed the title
[WIP] Science catalog full text crawlers[WIP] Full text crawlersAug 14, 2015
@mlep
mlep
commented
Aug 17, 2015

Copy link


Copy Markdown

Contributor
About "Do we want to include more Crawlers? If yes, which?":

You have a gold mine in the section "Search (and import) from my specific database, please." in the sorted list of feature requests (this morning email).
@stefan-kolb
stefan-kolb
commented
Aug 17, 2015

Copy link


Copy Markdown

MemberAuthor
@mlep Thanks 👍
@simonharrer
simonharrer
commented
Aug 18, 2015

Copy link


Copy Markdown

Contributor
Your feature works pretty well. But there are cases in which a file can be downloaded but it is not the file that is intended. Thus, there must be a way to preview the pdf before adding it.
@stefan-kolb
stefan-kolb
commented
Aug 18, 2015

Copy link


Copy Markdown

MemberAuthor
We need to evaluate how often this will happen. Can you give me an example?
I think with a reasonable amount of crawlers this will work with a very high accuracy.

I would rather have the user open and check the file and then delete it and manually download it if it is not the correct file, as this will happen very seldom. If we use your approach we need to preview every PDF that will be anyway in > 90% of the times the correct one.
@simonharrer
simonharrer
commented
Aug 18, 2015

Copy link


Copy Markdown

Contributor
Ok, deletion after download is OK when it is done easily. At the moment, I cannot easily delete the file entry from the bibtex entry AND also delete the file from the hard drive. When we fix this, we do not need the check feature in the first place. Good idea!
My example was: I wanted to download the conference article Barros, Service Interaction Patterns, but got the techrep instead. Another issue was that I got a PDF of slides in some language I do not understand, probably czech when attempting to download a very old version of the BPMN standard.
@stefan-kolbstefan-kolb
mentioned this pull request
Aug 18, 2015

Closed

@stefan-kolb
stefan-kolbforce-pushed
the
catalog-crawler
branch
2 times, most recently
from
43d0c7d to
3384c4cCompareAugust 18, 2015 12:49
@koppor
koppor
commented
Aug 18, 2015

Copy link


Copy Markdown

Member
Regarding, "clean all used fields from Latex stuff, e.g. {~ etc.", this should be one of the "Edit -> Cleanup entries" functionality, isn't it? If not, this functionality should be added there, too.
Regarding "check for duplicates", JabRef offers "Search -> find duplicates". This functionality should be reused here.
@koppor
koppor
commented
Aug 18, 2015

Copy link


Copy Markdown

Member
A similar approach might have been taken by Christoph Lehner: https://sourceforge.net/p/jabref/discussion/318824/thread/6e5fea64/ Is it possible to synchronize somehow?

I think, this makes https://github.com/wbrenna/LocalCopy obsolete, doesn't it?
@stefan-kolb
stefan-kolb
commented
Aug 19, 2015

Copy link


Copy Markdown

MemberAuthor
The approach of C.Lehner is the base repository https://github.com/lehner/LocalCopy.

This PR is a native replacement for this and all forks.

Also, we don't support plugins anymore.
@stefan-kolb
stefan-kolbforce-pushed
the
catalog-crawler
branch
5 times, most recently
from
4399958 to
5b6c7a4CompareAugust 24, 2015 15:51
@stefan-kolbstefan-kolb
added this to the v2.80 milestone
Aug 27, 2015
@minkkinenminkkinen
mentioned this pull request
Sep 18, 2015

Closed

@stefan-kolbstefan-kolb
mentioned this pull request
Sep 28, 2015
 Merged


This was referenced Oct 7, 2015

Closed


Open

@lenhardlenhard
mentioned this pull request
Oct 9, 2015

Closed

@lenhard
lenhard
commented
Oct 9, 2015

Copy link


Copy Markdown

Member
What is missing to get this PR functional and integrated into master?
@lenhardlenhard
assigned lenhard and simonharrer and unassigned lenhardOct 9, 2015
@simonharrer
simonharrer
commented
Oct 9, 2015

Copy link


Copy Markdown

Contributor
This is just too messy to understand right now. Fixing this would require fixing the separation of GUI Event Thread with other Code parts - a major effort.
The issue is how to implement swing actions that require multiple user interactions during their task taking place. Normally, this would require a nesting of swing worker classes, one for each step and starting the next step within the EDT update method.
What is more, sometimes SwingWorker, and sometimes the spin Framework is used. All of this makes this issue even more complicated.
Someone else may take a look if they see this issue more clearly than me.
@simonharrersimonharrer
removed their assignment
Oct 9, 2015
@lenhard
lenhard
commented
Oct 9, 2015

Copy link


Copy Markdown

Member
So you are saying that you will not complete the PR? I cannot see anybody else who will. There is no point of having this hanging around in limbo until it deprecates. So we can close this PR without merging and close related open issues as won't fix.
@simonharrer: Please confirm!
@simonharrer
simonharrer
commented
Oct 9, 2015

Copy link


Copy Markdown

Contributor
Merge as good enough.
@stefan-kolbstefan-kolb
changed the title
[WIP] Full text crawlersFull text crawlersOct 9, 2015
stefan-kolb added a commit
that referenced
this pull request
Oct 9, 2015
@stefan-kolb
Merge pull request#101from JabRef/catalog-crawler
18a6a0e
Full text crawlers
@stefan-kolb
stefan-kolb
merged commit 18a6a0e
into
masterOct 9, 2015
@stefan-kolb
stefan-kolb
deleted the

catalog-crawler

branch
October 9, 2015 13:34
@kopporkoppor
mentioned this pull request
Oct 10, 2015

Closed

@lecc0r
lecc0r
commented
Dec 1, 2015

Copy link


Copy Markdown

Hi, I've just tested the newly implemented "Full text article download" function, however for several articles it downloaded a version from ResearchGate but I want to have the one from Elsevier. Dependent who you are working for, you have special access privileges on certain publisher sites and it would be great to specify in the preferences which crawler the download function should prefer. Perhaps even define a priority list.
@stefan-kolb
stefan-kolb
commented
Dec 1, 2015

Copy link


Copy Markdown

MemberAuthor
There is a hard-coded priority list right know which prefers the official publishers over google scholar for example. If you want to file a new feature request or enhancement please create a separate issue.
@koppor
koppor
commented
Dec 1, 2015

Copy link


Copy Markdown

Member
Done at #435.
InAnYan added a commit
that referenced
this pull request
Aug 3, 2024
@InAnYan
Fix#99and partially#101
c245e73
github-merge-queueBot
pushed a commit
that referenced
this pull request
Aug 14, 2024
@InAnYan@koppor
AI chatting functionality (#11430)
8a0edc2
* Fix the code from code review
* Fix from code review and create new AiChatTabWorking
* Improve chat history storage code
* More fix from code review
* Remove obsolete parameter
* Add JavaDoc comment
* Fix checkstyle
* Fix JavaDoc
* Fix more checkstyle
* More checkstyle fixes
* Fix code changes
* Improve the PR
* Rework ADR-0031 to enable to use another option
* Add many LOGGEr.trace statements
* Change "message window" to "context window"
* Fix compiler errors
* Fix issue list index issue of langchain4j
* Fix lint issue
* Update 0031-store-chats-alongside-database.md
* More tracing
* Refine logging
* Remove closing of AiChatLanguageModel (because it's not closable)
* Use external package for OpenAI API connection
* Provide a custom executor for RetrievalAugmentor
* Fix shutdown issue (I hope)
* Refactor classes
* Change BibDatabaseChatHistoryFile
* Revert BibDatabaseChatHistoryFile to old version because of langchain4j
* Make round corners for chat messages
* Refactor embeddings generation
* Refactor embeddings generation
* Refactor embeddings generation
* Fix CHANGELOG.md
* Remove jpro-mdfx
* Add comment
* Fix localizations
* Fix checkstyle and remove OpenAI from PRIVACY.md
* Remove unnecessary comments
* Fix privacy notice UI
* Introduce new ApiKeyMissingComponent
* Thanks Tobiaz Diez for writing such a good EntryEditorTab class
* Fix InAnYan/jabref issues
* Merge `build.gradle` and `settings.gradle` from main branch
* Update ADRs
* Implement rethought ADR for chat history
* Use OpenAI embedding model
* Use Deep Java embedding model
* Remove old langchain4j embedding models
* Fix checkstyle errors
* Fix checkstyle and remove old dependencies
* Fixes from code review
* Restructure
* Fix checkstyle errors
* Add API base URL parameter
* Fix localization
* Fix from code review + ADR
* Something broken
* Now MistralAI and Hugging Face work
* Fix base URL for other LLM providers
* Fix base URL for other LLM providers
* Refactor MVStore usage
* Load embedding model in background
* Bump langchain4j version
* Fix bug
* Fix checkstyle and localization
* Implement summarization
* Fix checkstyle and localization
* Improve PrivacyNoticeComponent
* Fix from code review
* Update localization
* Wrap text
* Add padding
* Fix markdown
* Use stuff algorithm
* Add GPT-4o-mini
* Make chat model editable
* Update context window size and summarization
* Fix checkstyle
* Update PrivacyNoticeComponent.fxml
* Update AI summary tab
* Fix localization
* Change order so that there is no diff
* Reorrder dependencies
* Add missing CHANGELOG.md entry
* Refine ADR-0033
* Refine ADR0034
* Fix typos
* Refine ADR-0036
* Fix ADR-0037
* Fix title case
* Fix changes in module-info.java
* Readd removed requires org.apache.httpcomponents.core5.httpcore5
* Revert change in JabRefGUI to avoid conflicts
* Remove empty lines
* Reorder entries in JabRef_en.properties
* Simplify SummariesStorage (and add test)
* Use region/endregion
* Fix position of comment
* Add comment why the event bus is needed
* Do not show exception to the user - just that an error is occurred (saves %0 in localization)
* Use "URL %0" without colon (consistency)
* Fix typos
* History has to be kept
* Remove empty lines
* Fix language (hopefully)
* Compilefix
* Simplify BibDatabaseChatHistoryManager
* Fix from code review
* Fix issue #103
* Rework embeddings cache clearing
* Fix#99 and partially #101
* Partially fixing shutdown issues and UI progress monitor issue
* Add "requires scala.library" and add "region:" / "endregion"
* More grouping (move de.saxsys.mvvmfx.validation up)
* Add alphabetical hint
* FixInAnYan#101 and InAnYan#106
* Discard changes to settings.gradle
* FixInAnYan#105
* Follow-up fix for InAnYan#103
* Follow-up fix for InAnYan#103
* Remove obsolete class
* Partially fixInAnYan#98
* We do need dependencies to the AI providers, don't we?
* FixInAnYan#93
* Simplify code
* Partially fixInAnYan#92
* Fix checkstyle and localization
* Fix hyperlinks and text in ApiKeyMissingComponent
* Fixes from code review
* FixInAnYan#120
* Remove "X% work done" messages
* FixInAnYan#114
* Partially fixInAnYan#113
* Partially fixInAnYan#110
* FixInAnYan#110
* FixInAnYan#111
* Improve embedding model downloading notifications
* FixInAnYan#124
* FixInAnYan#122
* Fix wrong context window size when expert settings customization is turned off
* Attempt to fixInAnYan#95
* Finally fixInAnYan#105
* FixInAnYan#108
* Attempt to fixInAnYan#98
* Fix for InAnYan#104
* Fix for InAnYan#98
* Fix for InAnYan#95 (comment)
* Fix for InAnYan#98 (comment)
* Fix for InAnYan#126
* Fix for InAnYan#115
* Fix for InAnYan#113
* Fix for InAnYan#91
* Fix for InAnYan#121
* Fix for InAnYan#112 and InAnYan#116
* Fix for InAnYan#125
* Fixes from commit comments
* Fix for InAnYan#115
* Fix for InAnYan#120
* Fix for InAnYan#132
* Fix for InAnYan#132
* Fix for InAnYan#104
* Fix for InAnYan#118
* Fix for InAnYan#114
* Fix for InAnYan#104
* Store error messages in chat history
* Make error be a ChatMessageComponent
* Implement delete messages InAnYan#136
* Fix for InAnYan#118
* Fix for InAnYan#92
* Fix checkstyle and localization. And refactoring
* Fix for InAnYan#92
* Fix for InAnYan#139
* Show "Delete message" button only when necessary
* Fix for InAnYan#83
* Update src/main/java/org/jabref/logic/ai/AiService.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/logic/ai/chathistory/BibDatabaseChatHistoryManager.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/logic/ai/AiService.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/gui/Base.css
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/gui/Base.css
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Fix from code review
* Partial fix for InAnYan#125
* Update colors for error message
* Fix for InAnYan#145 and InAnYan#142
* Make progress for embedding model download
* Fix checkstyle and localization
* Add workaround to get FileHistoryMenuTest running again
* Small fixes
* Revert "Small fixes"
This reverts commit 85382a1.
* Introduce AiApiKeyProvider
* Fix IDE setup instructions
* Do not load API keys on startup
* Rely on keystore encryption
* Prevent mulitple rebuilds when muliple preferences are updated
* Fix localization to be more provider independent
* Fix method names
* Add poor man's solution to notify of API key changes
* Reduce calls to key store (and fix key saving)
* Fix for InAnYan#148 and partially InAnYan#146
* Revert "Fix for InAnYan#148 and partially InAnYan#146"
This reverts commit 5fa3bb5.
* Fix for scrolling down when deleting a message
* Sort EmbeddingModel enum variants
* Fix GenerateSummaryTask progress indication
* Fix dark mode
* Add notice for embedding models size
---------
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels

[outdated] type: feature

Projects


None yet


Milestone

v3.0

Development

Successfully merging this pull request may close these issues.

6 participants

@stefan-kolb@mlep@simonharrer@koppor@lenhard@lecc0r

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Full text crawlers by stefan-kolb · Pull Request #101 · JabRef/jabref · GitHub
Skip to content
Full text crawlers - #101
Merged
stefan-kolb merged 12 commits into
masterfrom 
catalog-crawler
Oct 9, 2015
Merged
Full text crawlers#101
stefan-kolb merged 12 commits into
masterfrom 
catalog-crawler
Conversation
@stefan-kolb
@stefan-kolbstefan-kolb
commented
Aug 14, 2015

Copy link


Copy Markdown

Member
This PR enables automatic PDF fulltext downloads.
Current catalogs:
  • ACS
  • arXiv
  • Springer
  • Sciencedirect (Elsevier)
  • Google Scholar
Questions:
  • Do we want to include more Crawlers? If yes, which?
  • Action was in the Tools menu. Imho it should be either auto downloaded or included with the downloador auto button in the detailed entry view.
TODO:
  •  Make it nonblocking and keep the progress bar!
  •  Add an option to delete the local file if a wrong file was downloaded
  •  Tests on Travis & CircleCI are blocked by Google by 403 Forbidden (Bot detection)
@stefan-kolbstefan-kolb
changed the title
Science catalog full text crawlers[WIP] Science catalog full text crawlersAug 14, 2015
@stefan-kolbstefan-kolb
changed the title
[WIP] Science catalog full text crawlers[WIP] Full text crawlersAug 14, 2015
@mlep
mlep
commented
Aug 17, 2015

Copy link


Copy Markdown

Contributor
About "Do we want to include more Crawlers? If yes, which?":

You have a gold mine in the section "Search (and import) from my specific database, please." in the sorted list of feature requests (this morning email).
@stefan-kolb
stefan-kolb
commented
Aug 17, 2015

Copy link


Copy Markdown

MemberAuthor
@mlep Thanks 👍
@simonharrer
simonharrer
commented
Aug 18, 2015

Copy link


Copy Markdown

Contributor
Your feature works pretty well. But there are cases in which a file can be downloaded but it is not the file that is intended. Thus, there must be a way to preview the pdf before adding it.
@stefan-kolb
stefan-kolb
commented
Aug 18, 2015

Copy link


Copy Markdown

MemberAuthor
We need to evaluate how often this will happen. Can you give me an example?
I think with a reasonable amount of crawlers this will work with a very high accuracy.

I would rather have the user open and check the file and then delete it and manually download it if it is not the correct file, as this will happen very seldom. If we use your approach we need to preview every PDF that will be anyway in > 90% of the times the correct one.
@simonharrer
simonharrer
commented
Aug 18, 2015

Copy link


Copy Markdown

Contributor
Ok, deletion after download is OK when it is done easily. At the moment, I cannot easily delete the file entry from the bibtex entry AND also delete the file from the hard drive. When we fix this, we do not need the check feature in the first place. Good idea!
My example was: I wanted to download the conference article Barros, Service Interaction Patterns, but got the techrep instead. Another issue was that I got a PDF of slides in some language I do not understand, probably czech when attempting to download a very old version of the BPMN standard.
@stefan-kolbstefan-kolb
mentioned this pull request
Aug 18, 2015

Closed

@stefan-kolb
stefan-kolbforce-pushed
the
catalog-crawler
branch
2 times, most recently
from
43d0c7d to
3384c4cCompareAugust 18, 2015 12:49
@koppor
koppor
commented
Aug 18, 2015

Copy link


Copy Markdown

Member
Regarding, "clean all used fields from Latex stuff, e.g. {~ etc.", this should be one of the "Edit -> Cleanup entries" functionality, isn't it? If not, this functionality should be added there, too.
Regarding "check for duplicates", JabRef offers "Search -> find duplicates". This functionality should be reused here.
@koppor
koppor
commented
Aug 18, 2015

Copy link


Copy Markdown

Member
A similar approach might have been taken by Christoph Lehner: https://sourceforge.net/p/jabref/discussion/318824/thread/6e5fea64/ Is it possible to synchronize somehow?

I think, this makes https://github.com/wbrenna/LocalCopy obsolete, doesn't it?
@stefan-kolb
stefan-kolb
commented
Aug 19, 2015

Copy link


Copy Markdown

MemberAuthor
The approach of C.Lehner is the base repository https://github.com/lehner/LocalCopy.

This PR is a native replacement for this and all forks.

Also, we don't support plugins anymore.
@stefan-kolb
stefan-kolbforce-pushed
the
catalog-crawler
branch
5 times, most recently
from
4399958 to
5b6c7a4CompareAugust 24, 2015 15:51
@stefan-kolbstefan-kolb
added this to the v2.80 milestone
Aug 27, 2015
@minkkinenminkkinen
mentioned this pull request
Sep 18, 2015

Closed

@stefan-kolbstefan-kolb
mentioned this pull request
Sep 28, 2015
 Merged


This was referenced Oct 7, 2015

Closed


Open

@lenhardlenhard
mentioned this pull request
Oct 9, 2015

Closed

@lenhard
lenhard
commented
Oct 9, 2015

Copy link


Copy Markdown

Member
What is missing to get this PR functional and integrated into master?
@lenhardlenhard
assigned lenhard and simonharrer and unassigned lenhardOct 9, 2015
@simonharrer
simonharrer
commented
Oct 9, 2015

Copy link


Copy Markdown

Contributor
This is just too messy to understand right now. Fixing this would require fixing the separation of GUI Event Thread with other Code parts - a major effort.
The issue is how to implement swing actions that require multiple user interactions during their task taking place. Normally, this would require a nesting of swing worker classes, one for each step and starting the next step within the EDT update method.
What is more, sometimes SwingWorker, and sometimes the spin Framework is used. All of this makes this issue even more complicated.
Someone else may take a look if they see this issue more clearly than me.
@simonharrersimonharrer
removed their assignment
Oct 9, 2015
@lenhard
lenhard
commented
Oct 9, 2015

Copy link


Copy Markdown

Member
So you are saying that you will not complete the PR? I cannot see anybody else who will. There is no point of having this hanging around in limbo until it deprecates. So we can close this PR without merging and close related open issues as won't fix.
@simonharrer: Please confirm!
@simonharrer
simonharrer
commented
Oct 9, 2015

Copy link


Copy Markdown

Contributor
Merge as good enough.
@stefan-kolbstefan-kolb
changed the title
[WIP] Full text crawlersFull text crawlersOct 9, 2015
stefan-kolb added a commit
that referenced
this pull request
Oct 9, 2015
@stefan-kolb
Merge pull request#101from JabRef/catalog-crawler
18a6a0e
Full text crawlers
@stefan-kolb
stefan-kolb
merged commit 18a6a0e
into
masterOct 9, 2015
@stefan-kolb
stefan-kolb
deleted the

catalog-crawler

branch
October 9, 2015 13:34
@kopporkoppor
mentioned this pull request
Oct 10, 2015

Closed

@lecc0r
lecc0r
commented
Dec 1, 2015

Copy link


Copy Markdown

Hi, I've just tested the newly implemented "Full text article download" function, however for several articles it downloaded a version from ResearchGate but I want to have the one from Elsevier. Dependent who you are working for, you have special access privileges on certain publisher sites and it would be great to specify in the preferences which crawler the download function should prefer. Perhaps even define a priority list.
@stefan-kolb
stefan-kolb
commented
Dec 1, 2015

Copy link


Copy Markdown

MemberAuthor
There is a hard-coded priority list right know which prefers the official publishers over google scholar for example. If you want to file a new feature request or enhancement please create a separate issue.
@koppor
koppor
commented
Dec 1, 2015

Copy link


Copy Markdown

Member
Done at #435.
InAnYan added a commit
that referenced
this pull request
Aug 3, 2024
@InAnYan
Fix#99and partially#101
c245e73
github-merge-queueBot
pushed a commit
that referenced
this pull request
Aug 14, 2024
@InAnYan@koppor
AI chatting functionality (#11430)
8a0edc2
* Fix the code from code review
* Fix from code review and create new AiChatTabWorking
* Improve chat history storage code
* More fix from code review
* Remove obsolete parameter
* Add JavaDoc comment
* Fix checkstyle
* Fix JavaDoc
* Fix more checkstyle
* More checkstyle fixes
* Fix code changes
* Improve the PR
* Rework ADR-0031 to enable to use another option
* Add many LOGGEr.trace statements
* Change "message window" to "context window"
* Fix compiler errors
* Fix issue list index issue of langchain4j
* Fix lint issue
* Update 0031-store-chats-alongside-database.md
* More tracing
* Refine logging
* Remove closing of AiChatLanguageModel (because it's not closable)
* Use external package for OpenAI API connection
* Provide a custom executor for RetrievalAugmentor
* Fix shutdown issue (I hope)
* Refactor classes
* Change BibDatabaseChatHistoryFile
* Revert BibDatabaseChatHistoryFile to old version because of langchain4j
* Make round corners for chat messages
* Refactor embeddings generation
* Refactor embeddings generation
* Refactor embeddings generation
* Fix CHANGELOG.md
* Remove jpro-mdfx
* Add comment
* Fix localizations
* Fix checkstyle and remove OpenAI from PRIVACY.md
* Remove unnecessary comments
* Fix privacy notice UI
* Introduce new ApiKeyMissingComponent
* Thanks Tobiaz Diez for writing such a good EntryEditorTab class
* Fix InAnYan/jabref issues
* Merge `build.gradle` and `settings.gradle` from main branch
* Update ADRs
* Implement rethought ADR for chat history
* Use OpenAI embedding model
* Use Deep Java embedding model
* Remove old langchain4j embedding models
* Fix checkstyle errors
* Fix checkstyle and remove old dependencies
* Fixes from code review
* Restructure
* Fix checkstyle errors
* Add API base URL parameter
* Fix localization
* Fix from code review + ADR
* Something broken
* Now MistralAI and Hugging Face work
* Fix base URL for other LLM providers
* Fix base URL for other LLM providers
* Refactor MVStore usage
* Load embedding model in background
* Bump langchain4j version
* Fix bug
* Fix checkstyle and localization
* Implement summarization
* Fix checkstyle and localization
* Improve PrivacyNoticeComponent
* Fix from code review
* Update localization
* Wrap text
* Add padding
* Fix markdown
* Use stuff algorithm
* Add GPT-4o-mini
* Make chat model editable
* Update context window size and summarization
* Fix checkstyle
* Update PrivacyNoticeComponent.fxml
* Update AI summary tab
* Fix localization
* Change order so that there is no diff
* Reorrder dependencies
* Add missing CHANGELOG.md entry
* Refine ADR-0033
* Refine ADR0034
* Fix typos
* Refine ADR-0036
* Fix ADR-0037
* Fix title case
* Fix changes in module-info.java
* Readd removed requires org.apache.httpcomponents.core5.httpcore5
* Revert change in JabRefGUI to avoid conflicts
* Remove empty lines
* Reorder entries in JabRef_en.properties
* Simplify SummariesStorage (and add test)
* Use region/endregion
* Fix position of comment
* Add comment why the event bus is needed
* Do not show exception to the user - just that an error is occurred (saves %0 in localization)
* Use "URL %0" without colon (consistency)
* Fix typos
* History has to be kept
* Remove empty lines
* Fix language (hopefully)
* Compilefix
* Simplify BibDatabaseChatHistoryManager
* Fix from code review
* Fix issue #103
* Rework embeddings cache clearing
* Fix#99 and partially #101
* Partially fixing shutdown issues and UI progress monitor issue
* Add "requires scala.library" and add "region:" / "endregion"
* More grouping (move de.saxsys.mvvmfx.validation up)
* Add alphabetical hint
* FixInAnYan#101 and InAnYan#106
* Discard changes to settings.gradle
* FixInAnYan#105
* Follow-up fix for InAnYan#103
* Follow-up fix for InAnYan#103
* Remove obsolete class
* Partially fixInAnYan#98
* We do need dependencies to the AI providers, don't we?
* FixInAnYan#93
* Simplify code
* Partially fixInAnYan#92
* Fix checkstyle and localization
* Fix hyperlinks and text in ApiKeyMissingComponent
* Fixes from code review
* FixInAnYan#120
* Remove "X% work done" messages
* FixInAnYan#114
* Partially fixInAnYan#113
* Partially fixInAnYan#110
* FixInAnYan#110
* FixInAnYan#111
* Improve embedding model downloading notifications
* FixInAnYan#124
* FixInAnYan#122
* Fix wrong context window size when expert settings customization is turned off
* Attempt to fixInAnYan#95
* Finally fixInAnYan#105
* FixInAnYan#108
* Attempt to fixInAnYan#98
* Fix for InAnYan#104
* Fix for InAnYan#98
* Fix for InAnYan#95 (comment)
* Fix for InAnYan#98 (comment)
* Fix for InAnYan#126
* Fix for InAnYan#115
* Fix for InAnYan#113
* Fix for InAnYan#91
* Fix for InAnYan#121
* Fix for InAnYan#112 and InAnYan#116
* Fix for InAnYan#125
* Fixes from commit comments
* Fix for InAnYan#115
* Fix for InAnYan#120
* Fix for InAnYan#132
* Fix for InAnYan#132
* Fix for InAnYan#104
* Fix for InAnYan#118
* Fix for InAnYan#114
* Fix for InAnYan#104
* Store error messages in chat history
* Make error be a ChatMessageComponent
* Implement delete messages InAnYan#136
* Fix for InAnYan#118
* Fix for InAnYan#92
* Fix checkstyle and localization. And refactoring
* Fix for InAnYan#92
* Fix for InAnYan#139
* Show "Delete message" button only when necessary
* Fix for InAnYan#83
* Update src/main/java/org/jabref/logic/ai/AiService.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/logic/ai/chathistory/BibDatabaseChatHistoryManager.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/logic/ai/AiService.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/gui/Base.css
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/gui/Base.css
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Fix from code review
* Partial fix for InAnYan#125
* Update colors for error message
* Fix for InAnYan#145 and InAnYan#142
* Make progress for embedding model download
* Fix checkstyle and localization
* Add workaround to get FileHistoryMenuTest running again
* Small fixes
* Revert "Small fixes"
This reverts commit 85382a1.
* Introduce AiApiKeyProvider
* Fix IDE setup instructions
* Do not load API keys on startup
* Rely on keystore encryption
* Prevent mulitple rebuilds when muliple preferences are updated
* Fix localization to be more provider independent
* Fix method names
* Add poor man's solution to notify of API key changes
* Reduce calls to key store (and fix key saving)
* Fix for InAnYan#148 and partially InAnYan#146
* Revert "Fix for InAnYan#148 and partially InAnYan#146"
This reverts commit 5fa3bb5.
* Fix for scrolling down when deleting a message
* Sort EmbeddingModel enum variants
* Fix GenerateSummaryTask progress indication
* Fix dark mode
* Add notice for embedding models size
---------
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels

[outdated] type: feature

Projects


None yet


Milestone

v3.0

Development

Successfully merging this pull request may close these issues.

6 participants

@stefan-kolb@mlep@simonharrer@koppor@lenhard@lecc0r

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Full text crawlers by stefan-kolb · Pull Request #101 · JabRef/jabref · GitHub
Skip to content
Full text crawlers - #101
Merged
stefan-kolb merged 12 commits into
masterfrom 
catalog-crawler
Oct 9, 2015
Merged
Full text crawlers#101
stefan-kolb merged 12 commits into
masterfrom 
catalog-crawler
Conversation
@stefan-kolb
@stefan-kolbstefan-kolb
commented
Aug 14, 2015

Copy link


Copy Markdown

Member
This PR enables automatic PDF fulltext downloads.
Current catalogs:
  • ACS
  • arXiv
  • Springer
  • Sciencedirect (Elsevier)
  • Google Scholar
Questions:
  • Do we want to include more Crawlers? If yes, which?
  • Action was in the Tools menu. Imho it should be either auto downloaded or included with the downloador auto button in the detailed entry view.
TODO:
  •  Make it nonblocking and keep the progress bar!
  •  Add an option to delete the local file if a wrong file was downloaded
  •  Tests on Travis & CircleCI are blocked by Google by 403 Forbidden (Bot detection)
@stefan-kolbstefan-kolb
changed the title
Science catalog full text crawlers[WIP] Science catalog full text crawlersAug 14, 2015
@stefan-kolbstefan-kolb
changed the title
[WIP] Science catalog full text crawlers[WIP] Full text crawlersAug 14, 2015
@mlep
mlep
commented
Aug 17, 2015

Copy link


Copy Markdown

Contributor
About "Do we want to include more Crawlers? If yes, which?":

You have a gold mine in the section "Search (and import) from my specific database, please." in the sorted list of feature requests (this morning email).
@stefan-kolb
stefan-kolb
commented
Aug 17, 2015

Copy link


Copy Markdown

MemberAuthor
@mlep Thanks 👍
@simonharrer
simonharrer
commented
Aug 18, 2015

Copy link


Copy Markdown

Contributor
Your feature works pretty well. But there are cases in which a file can be downloaded but it is not the file that is intended. Thus, there must be a way to preview the pdf before adding it.
@stefan-kolb
stefan-kolb
commented
Aug 18, 2015

Copy link


Copy Markdown

MemberAuthor
We need to evaluate how often this will happen. Can you give me an example?
I think with a reasonable amount of crawlers this will work with a very high accuracy.

I would rather have the user open and check the file and then delete it and manually download it if it is not the correct file, as this will happen very seldom. If we use your approach we need to preview every PDF that will be anyway in > 90% of the times the correct one.
@simonharrer
simonharrer
commented
Aug 18, 2015

Copy link


Copy Markdown

Contributor
Ok, deletion after download is OK when it is done easily. At the moment, I cannot easily delete the file entry from the bibtex entry AND also delete the file from the hard drive. When we fix this, we do not need the check feature in the first place. Good idea!
My example was: I wanted to download the conference article Barros, Service Interaction Patterns, but got the techrep instead. Another issue was that I got a PDF of slides in some language I do not understand, probably czech when attempting to download a very old version of the BPMN standard.
@stefan-kolbstefan-kolb
mentioned this pull request
Aug 18, 2015

Closed

@stefan-kolb
stefan-kolbforce-pushed
the
catalog-crawler
branch
2 times, most recently
from
43d0c7d to
3384c4cCompareAugust 18, 2015 12:49
@koppor
koppor
commented
Aug 18, 2015

Copy link


Copy Markdown

Member
Regarding, "clean all used fields from Latex stuff, e.g. {~ etc.", this should be one of the "Edit -> Cleanup entries" functionality, isn't it? If not, this functionality should be added there, too.
Regarding "check for duplicates", JabRef offers "Search -> find duplicates". This functionality should be reused here.
@koppor
koppor
commented
Aug 18, 2015

Copy link


Copy Markdown

Member
A similar approach might have been taken by Christoph Lehner: https://sourceforge.net/p/jabref/discussion/318824/thread/6e5fea64/ Is it possible to synchronize somehow?

I think, this makes https://github.com/wbrenna/LocalCopy obsolete, doesn't it?
@stefan-kolb
stefan-kolb
commented
Aug 19, 2015

Copy link


Copy Markdown

MemberAuthor
The approach of C.Lehner is the base repository https://github.com/lehner/LocalCopy.

This PR is a native replacement for this and all forks.

Also, we don't support plugins anymore.
@stefan-kolb
stefan-kolbforce-pushed
the
catalog-crawler
branch
5 times, most recently
from
4399958 to
5b6c7a4CompareAugust 24, 2015 15:51
@stefan-kolbstefan-kolb
added this to the v2.80 milestone
Aug 27, 2015
@minkkinenminkkinen
mentioned this pull request
Sep 18, 2015

Closed

@stefan-kolbstefan-kolb
mentioned this pull request
Sep 28, 2015
 Merged


This was referenced Oct 7, 2015

Closed


Open

@lenhardlenhard
mentioned this pull request
Oct 9, 2015

Closed

@lenhard
lenhard
commented
Oct 9, 2015

Copy link


Copy Markdown

Member
What is missing to get this PR functional and integrated into master?
@lenhardlenhard
assigned lenhard and simonharrer and unassigned lenhardOct 9, 2015
@simonharrer
simonharrer
commented
Oct 9, 2015

Copy link


Copy Markdown

Contributor
This is just too messy to understand right now. Fixing this would require fixing the separation of GUI Event Thread with other Code parts - a major effort.
The issue is how to implement swing actions that require multiple user interactions during their task taking place. Normally, this would require a nesting of swing worker classes, one for each step and starting the next step within the EDT update method.
What is more, sometimes SwingWorker, and sometimes the spin Framework is used. All of this makes this issue even more complicated.
Someone else may take a look if they see this issue more clearly than me.
@simonharrersimonharrer
removed their assignment
Oct 9, 2015
@lenhard
lenhard
commented
Oct 9, 2015

Copy link


Copy Markdown

Member
So you are saying that you will not complete the PR? I cannot see anybody else who will. There is no point of having this hanging around in limbo until it deprecates. So we can close this PR without merging and close related open issues as won't fix.
@simonharrer: Please confirm!
@simonharrer
simonharrer
commented
Oct 9, 2015

Copy link


Copy Markdown

Contributor
Merge as good enough.
@stefan-kolbstefan-kolb
changed the title
[WIP] Full text crawlersFull text crawlersOct 9, 2015
stefan-kolb added a commit
that referenced
this pull request
Oct 9, 2015
@stefan-kolb
Merge pull request#101from JabRef/catalog-crawler
18a6a0e
Full text crawlers
@stefan-kolb
stefan-kolb
merged commit 18a6a0e
into
masterOct 9, 2015
@stefan-kolb
stefan-kolb
deleted the

catalog-crawler

branch
October 9, 2015 13:34
@kopporkoppor
mentioned this pull request
Oct 10, 2015

Closed

@lecc0r
lecc0r
commented
Dec 1, 2015

Copy link


Copy Markdown

Hi, I've just tested the newly implemented "Full text article download" function, however for several articles it downloaded a version from ResearchGate but I want to have the one from Elsevier. Dependent who you are working for, you have special access privileges on certain publisher sites and it would be great to specify in the preferences which crawler the download function should prefer. Perhaps even define a priority list.
@stefan-kolb
stefan-kolb
commented
Dec 1, 2015

Copy link


Copy Markdown

MemberAuthor
There is a hard-coded priority list right know which prefers the official publishers over google scholar for example. If you want to file a new feature request or enhancement please create a separate issue.
@koppor
koppor
commented
Dec 1, 2015

Copy link


Copy Markdown

Member
Done at #435.
InAnYan added a commit
that referenced
this pull request
Aug 3, 2024
@InAnYan
Fix#99and partially#101
c245e73
github-merge-queueBot
pushed a commit
that referenced
this pull request
Aug 14, 2024
@InAnYan@koppor
AI chatting functionality (#11430)
8a0edc2
* Fix the code from code review
* Fix from code review and create new AiChatTabWorking
* Improve chat history storage code
* More fix from code review
* Remove obsolete parameter
* Add JavaDoc comment
* Fix checkstyle
* Fix JavaDoc
* Fix more checkstyle
* More checkstyle fixes
* Fix code changes
* Improve the PR
* Rework ADR-0031 to enable to use another option
* Add many LOGGEr.trace statements
* Change "message window" to "context window"
* Fix compiler errors
* Fix issue list index issue of langchain4j
* Fix lint issue
* Update 0031-store-chats-alongside-database.md
* More tracing
* Refine logging
* Remove closing of AiChatLanguageModel (because it's not closable)
* Use external package for OpenAI API connection
* Provide a custom executor for RetrievalAugmentor
* Fix shutdown issue (I hope)
* Refactor classes
* Change BibDatabaseChatHistoryFile
* Revert BibDatabaseChatHistoryFile to old version because of langchain4j
* Make round corners for chat messages
* Refactor embeddings generation
* Refactor embeddings generation
* Refactor embeddings generation
* Fix CHANGELOG.md
* Remove jpro-mdfx
* Add comment
* Fix localizations
* Fix checkstyle and remove OpenAI from PRIVACY.md
* Remove unnecessary comments
* Fix privacy notice UI
* Introduce new ApiKeyMissingComponent
* Thanks Tobiaz Diez for writing such a good EntryEditorTab class
* Fix InAnYan/jabref issues
* Merge `build.gradle` and `settings.gradle` from main branch
* Update ADRs
* Implement rethought ADR for chat history
* Use OpenAI embedding model
* Use Deep Java embedding model
* Remove old langchain4j embedding models
* Fix checkstyle errors
* Fix checkstyle and remove old dependencies
* Fixes from code review
* Restructure
* Fix checkstyle errors
* Add API base URL parameter
* Fix localization
* Fix from code review + ADR
* Something broken
* Now MistralAI and Hugging Face work
* Fix base URL for other LLM providers
* Fix base URL for other LLM providers
* Refactor MVStore usage
* Load embedding model in background
* Bump langchain4j version
* Fix bug
* Fix checkstyle and localization
* Implement summarization
* Fix checkstyle and localization
* Improve PrivacyNoticeComponent
* Fix from code review
* Update localization
* Wrap text
* Add padding
* Fix markdown
* Use stuff algorithm
* Add GPT-4o-mini
* Make chat model editable
* Update context window size and summarization
* Fix checkstyle
* Update PrivacyNoticeComponent.fxml
* Update AI summary tab
* Fix localization
* Change order so that there is no diff
* Reorrder dependencies
* Add missing CHANGELOG.md entry
* Refine ADR-0033
* Refine ADR0034
* Fix typos
* Refine ADR-0036
* Fix ADR-0037
* Fix title case
* Fix changes in module-info.java
* Readd removed requires org.apache.httpcomponents.core5.httpcore5
* Revert change in JabRefGUI to avoid conflicts
* Remove empty lines
* Reorder entries in JabRef_en.properties
* Simplify SummariesStorage (and add test)
* Use region/endregion
* Fix position of comment
* Add comment why the event bus is needed
* Do not show exception to the user - just that an error is occurred (saves %0 in localization)
* Use "URL %0" without colon (consistency)
* Fix typos
* History has to be kept
* Remove empty lines
* Fix language (hopefully)
* Compilefix
* Simplify BibDatabaseChatHistoryManager
* Fix from code review
* Fix issue #103
* Rework embeddings cache clearing
* Fix#99 and partially #101
* Partially fixing shutdown issues and UI progress monitor issue
* Add "requires scala.library" and add "region:" / "endregion"
* More grouping (move de.saxsys.mvvmfx.validation up)
* Add alphabetical hint
* FixInAnYan#101 and InAnYan#106
* Discard changes to settings.gradle
* FixInAnYan#105
* Follow-up fix for InAnYan#103
* Follow-up fix for InAnYan#103
* Remove obsolete class
* Partially fixInAnYan#98
* We do need dependencies to the AI providers, don't we?
* FixInAnYan#93
* Simplify code
* Partially fixInAnYan#92
* Fix checkstyle and localization
* Fix hyperlinks and text in ApiKeyMissingComponent
* Fixes from code review
* FixInAnYan#120
* Remove "X% work done" messages
* FixInAnYan#114
* Partially fixInAnYan#113
* Partially fixInAnYan#110
* FixInAnYan#110
* FixInAnYan#111
* Improve embedding model downloading notifications
* FixInAnYan#124
* FixInAnYan#122
* Fix wrong context window size when expert settings customization is turned off
* Attempt to fixInAnYan#95
* Finally fixInAnYan#105
* FixInAnYan#108
* Attempt to fixInAnYan#98
* Fix for InAnYan#104
* Fix for InAnYan#98
* Fix for InAnYan#95 (comment)
* Fix for InAnYan#98 (comment)
* Fix for InAnYan#126
* Fix for InAnYan#115
* Fix for InAnYan#113
* Fix for InAnYan#91
* Fix for InAnYan#121
* Fix for InAnYan#112 and InAnYan#116
* Fix for InAnYan#125
* Fixes from commit comments
* Fix for InAnYan#115
* Fix for InAnYan#120
* Fix for InAnYan#132
* Fix for InAnYan#132
* Fix for InAnYan#104
* Fix for InAnYan#118
* Fix for InAnYan#114
* Fix for InAnYan#104
* Store error messages in chat history
* Make error be a ChatMessageComponent
* Implement delete messages InAnYan#136
* Fix for InAnYan#118
* Fix for InAnYan#92
* Fix checkstyle and localization. And refactoring
* Fix for InAnYan#92
* Fix for InAnYan#139
* Show "Delete message" button only when necessary
* Fix for InAnYan#83
* Update src/main/java/org/jabref/logic/ai/AiService.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/logic/ai/chathistory/BibDatabaseChatHistoryManager.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/logic/ai/AiService.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/gui/Base.css
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/gui/Base.css
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Fix from code review
* Partial fix for InAnYan#125
* Update colors for error message
* Fix for InAnYan#145 and InAnYan#142
* Make progress for embedding model download
* Fix checkstyle and localization
* Add workaround to get FileHistoryMenuTest running again
* Small fixes
* Revert "Small fixes"
This reverts commit 85382a1.
* Introduce AiApiKeyProvider
* Fix IDE setup instructions
* Do not load API keys on startup
* Rely on keystore encryption
* Prevent mulitple rebuilds when muliple preferences are updated
* Fix localization to be more provider independent
* Fix method names
* Add poor man's solution to notify of API key changes
* Reduce calls to key store (and fix key saving)
* Fix for InAnYan#148 and partially InAnYan#146
* Revert "Fix for InAnYan#148 and partially InAnYan#146"
This reverts commit 5fa3bb5.
* Fix for scrolling down when deleting a message
* Sort EmbeddingModel enum variants
* Fix GenerateSummaryTask progress indication
* Fix dark mode
* Add notice for embedding models size
---------
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels

[outdated] type: feature

Projects


None yet


Milestone

v3.0

Development

Successfully merging this pull request may close these issues.

6 participants

@stefan-kolb@mlep@simonharrer@koppor@lenhard@lecc0r

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
Full text crawlers by stefan-kolb · Pull Request #101 · JabRef/jabref · GitHub
Skip to content
Full text crawlers - #101
Merged
stefan-kolb merged 12 commits into
masterfrom 
catalog-crawler
Oct 9, 2015
Merged
Full text crawlers#101
stefan-kolb merged 12 commits into
masterfrom 
catalog-crawler
Conversation
@stefan-kolb
@stefan-kolbstefan-kolb
commented
Aug 14, 2015

Copy link


Copy Markdown

Member
This PR enables automatic PDF fulltext downloads.
Current catalogs:
  • ACS
  • arXiv
  • Springer
  • Sciencedirect (Elsevier)
  • Google Scholar
Questions:
  • Do we want to include more Crawlers? If yes, which?
  • Action was in the Tools menu. Imho it should be either auto downloaded or included with the downloador auto button in the detailed entry view.
TODO:
  •  Make it nonblocking and keep the progress bar!
  •  Add an option to delete the local file if a wrong file was downloaded
  •  Tests on Travis & CircleCI are blocked by Google by 403 Forbidden (Bot detection)
@stefan-kolbstefan-kolb
changed the title
Science catalog full text crawlers[WIP] Science catalog full text crawlersAug 14, 2015
@stefan-kolbstefan-kolb
changed the title
[WIP] Science catalog full text crawlers[WIP] Full text crawlersAug 14, 2015
@mlep
mlep
commented
Aug 17, 2015

Copy link


Copy Markdown

Contributor
About "Do we want to include more Crawlers? If yes, which?":

You have a gold mine in the section "Search (and import) from my specific database, please." in the sorted list of feature requests (this morning email).
@stefan-kolb
stefan-kolb
commented
Aug 17, 2015

Copy link


Copy Markdown

MemberAuthor
@mlep Thanks 👍
@simonharrer
simonharrer
commented
Aug 18, 2015

Copy link


Copy Markdown

Contributor
Your feature works pretty well. But there are cases in which a file can be downloaded but it is not the file that is intended. Thus, there must be a way to preview the pdf before adding it.
@stefan-kolb
stefan-kolb
commented
Aug 18, 2015

Copy link


Copy Markdown

MemberAuthor
We need to evaluate how often this will happen. Can you give me an example?
I think with a reasonable amount of crawlers this will work with a very high accuracy.

I would rather have the user open and check the file and then delete it and manually download it if it is not the correct file, as this will happen very seldom. If we use your approach we need to preview every PDF that will be anyway in > 90% of the times the correct one.
@simonharrer
simonharrer
commented
Aug 18, 2015

Copy link


Copy Markdown

Contributor
Ok, deletion after download is OK when it is done easily. At the moment, I cannot easily delete the file entry from the bibtex entry AND also delete the file from the hard drive. When we fix this, we do not need the check feature in the first place. Good idea!
My example was: I wanted to download the conference article Barros, Service Interaction Patterns, but got the techrep instead. Another issue was that I got a PDF of slides in some language I do not understand, probably czech when attempting to download a very old version of the BPMN standard.
@stefan-kolbstefan-kolb
mentioned this pull request
Aug 18, 2015

Closed

@stefan-kolb
stefan-kolbforce-pushed
the
catalog-crawler
branch
2 times, most recently
from
43d0c7d to
3384c4cCompareAugust 18, 2015 12:49
@koppor
koppor
commented
Aug 18, 2015

Copy link


Copy Markdown

Member
Regarding, "clean all used fields from Latex stuff, e.g. {~ etc.", this should be one of the "Edit -> Cleanup entries" functionality, isn't it? If not, this functionality should be added there, too.
Regarding "check for duplicates", JabRef offers "Search -> find duplicates". This functionality should be reused here.
@koppor
koppor
commented
Aug 18, 2015

Copy link


Copy Markdown

Member
A similar approach might have been taken by Christoph Lehner: https://sourceforge.net/p/jabref/discussion/318824/thread/6e5fea64/ Is it possible to synchronize somehow?

I think, this makes https://github.com/wbrenna/LocalCopy obsolete, doesn't it?
@stefan-kolb
stefan-kolb
commented
Aug 19, 2015

Copy link


Copy Markdown

MemberAuthor
The approach of C.Lehner is the base repository https://github.com/lehner/LocalCopy.

This PR is a native replacement for this and all forks.

Also, we don't support plugins anymore.
@stefan-kolb
stefan-kolbforce-pushed
the
catalog-crawler
branch
5 times, most recently
from
4399958 to
5b6c7a4CompareAugust 24, 2015 15:51
@stefan-kolbstefan-kolb
added this to the v2.80 milestone
Aug 27, 2015
@minkkinenminkkinen
mentioned this pull request
Sep 18, 2015

Closed

@stefan-kolbstefan-kolb
mentioned this pull request
Sep 28, 2015
 Merged


This was referenced Oct 7, 2015

Closed


Open

@lenhardlenhard
mentioned this pull request
Oct 9, 2015

Closed

@lenhard
lenhard
commented
Oct 9, 2015

Copy link


Copy Markdown

Member
What is missing to get this PR functional and integrated into master?
@lenhardlenhard
assigned lenhard and simonharrer and unassigned lenhardOct 9, 2015
@simonharrer
simonharrer
commented
Oct 9, 2015

Copy link


Copy Markdown

Contributor
This is just too messy to understand right now. Fixing this would require fixing the separation of GUI Event Thread with other Code parts - a major effort.
The issue is how to implement swing actions that require multiple user interactions during their task taking place. Normally, this would require a nesting of swing worker classes, one for each step and starting the next step within the EDT update method.
What is more, sometimes SwingWorker, and sometimes the spin Framework is used. All of this makes this issue even more complicated.
Someone else may take a look if they see this issue more clearly than me.
@simonharrersimonharrer
removed their assignment
Oct 9, 2015
@lenhard
lenhard
commented
Oct 9, 2015

Copy link


Copy Markdown

Member
So you are saying that you will not complete the PR? I cannot see anybody else who will. There is no point of having this hanging around in limbo until it deprecates. So we can close this PR without merging and close related open issues as won't fix.
@simonharrer: Please confirm!
@simonharrer
simonharrer
commented
Oct 9, 2015

Copy link


Copy Markdown

Contributor
Merge as good enough.
@stefan-kolbstefan-kolb
changed the title
[WIP] Full text crawlersFull text crawlersOct 9, 2015
stefan-kolb added a commit
that referenced
this pull request
Oct 9, 2015
@stefan-kolb
Merge pull request#101from JabRef/catalog-crawler
18a6a0e
Full text crawlers
@stefan-kolb
stefan-kolb
merged commit 18a6a0e
into
masterOct 9, 2015
@stefan-kolb
stefan-kolb
deleted the

catalog-crawler

branch
October 9, 2015 13:34
@kopporkoppor
mentioned this pull request
Oct 10, 2015

Closed

@lecc0r
lecc0r
commented
Dec 1, 2015

Copy link


Copy Markdown

Hi, I've just tested the newly implemented "Full text article download" function, however for several articles it downloaded a version from ResearchGate but I want to have the one from Elsevier. Dependent who you are working for, you have special access privileges on certain publisher sites and it would be great to specify in the preferences which crawler the download function should prefer. Perhaps even define a priority list.
@stefan-kolb
stefan-kolb
commented
Dec 1, 2015

Copy link


Copy Markdown

MemberAuthor
There is a hard-coded priority list right know which prefers the official publishers over google scholar for example. If you want to file a new feature request or enhancement please create a separate issue.
@koppor
koppor
commented
Dec 1, 2015

Copy link


Copy Markdown

Member
Done at #435.
InAnYan added a commit
that referenced
this pull request
Aug 3, 2024
@InAnYan
Fix#99and partially#101
c245e73
github-merge-queueBot
pushed a commit
that referenced
this pull request
Aug 14, 2024
@InAnYan@koppor
AI chatting functionality (#11430)
8a0edc2
* Fix the code from code review
* Fix from code review and create new AiChatTabWorking
* Improve chat history storage code
* More fix from code review
* Remove obsolete parameter
* Add JavaDoc comment
* Fix checkstyle
* Fix JavaDoc
* Fix more checkstyle
* More checkstyle fixes
* Fix code changes
* Improve the PR
* Rework ADR-0031 to enable to use another option
* Add many LOGGEr.trace statements
* Change "message window" to "context window"
* Fix compiler errors
* Fix issue list index issue of langchain4j
* Fix lint issue
* Update 0031-store-chats-alongside-database.md
* More tracing
* Refine logging
* Remove closing of AiChatLanguageModel (because it's not closable)
* Use external package for OpenAI API connection
* Provide a custom executor for RetrievalAugmentor
* Fix shutdown issue (I hope)
* Refactor classes
* Change BibDatabaseChatHistoryFile
* Revert BibDatabaseChatHistoryFile to old version because of langchain4j
* Make round corners for chat messages
* Refactor embeddings generation
* Refactor embeddings generation
* Refactor embeddings generation
* Fix CHANGELOG.md
* Remove jpro-mdfx
* Add comment
* Fix localizations
* Fix checkstyle and remove OpenAI from PRIVACY.md
* Remove unnecessary comments
* Fix privacy notice UI
* Introduce new ApiKeyMissingComponent
* Thanks Tobiaz Diez for writing such a good EntryEditorTab class
* Fix InAnYan/jabref issues
* Merge `build.gradle` and `settings.gradle` from main branch
* Update ADRs
* Implement rethought ADR for chat history
* Use OpenAI embedding model
* Use Deep Java embedding model
* Remove old langchain4j embedding models
* Fix checkstyle errors
* Fix checkstyle and remove old dependencies
* Fixes from code review
* Restructure
* Fix checkstyle errors
* Add API base URL parameter
* Fix localization
* Fix from code review + ADR
* Something broken
* Now MistralAI and Hugging Face work
* Fix base URL for other LLM providers
* Fix base URL for other LLM providers
* Refactor MVStore usage
* Load embedding model in background
* Bump langchain4j version
* Fix bug
* Fix checkstyle and localization
* Implement summarization
* Fix checkstyle and localization
* Improve PrivacyNoticeComponent
* Fix from code review
* Update localization
* Wrap text
* Add padding
* Fix markdown
* Use stuff algorithm
* Add GPT-4o-mini
* Make chat model editable
* Update context window size and summarization
* Fix checkstyle
* Update PrivacyNoticeComponent.fxml
* Update AI summary tab
* Fix localization
* Change order so that there is no diff
* Reorrder dependencies
* Add missing CHANGELOG.md entry
* Refine ADR-0033
* Refine ADR0034
* Fix typos
* Refine ADR-0036
* Fix ADR-0037
* Fix title case
* Fix changes in module-info.java
* Readd removed requires org.apache.httpcomponents.core5.httpcore5
* Revert change in JabRefGUI to avoid conflicts
* Remove empty lines
* Reorder entries in JabRef_en.properties
* Simplify SummariesStorage (and add test)
* Use region/endregion
* Fix position of comment
* Add comment why the event bus is needed
* Do not show exception to the user - just that an error is occurred (saves %0 in localization)
* Use "URL %0" without colon (consistency)
* Fix typos
* History has to be kept
* Remove empty lines
* Fix language (hopefully)
* Compilefix
* Simplify BibDatabaseChatHistoryManager
* Fix from code review
* Fix issue #103
* Rework embeddings cache clearing
* Fix#99 and partially #101
* Partially fixing shutdown issues and UI progress monitor issue
* Add "requires scala.library" and add "region:" / "endregion"
* More grouping (move de.saxsys.mvvmfx.validation up)
* Add alphabetical hint
* FixInAnYan#101 and InAnYan#106
* Discard changes to settings.gradle
* FixInAnYan#105
* Follow-up fix for InAnYan#103
* Follow-up fix for InAnYan#103
* Remove obsolete class
* Partially fixInAnYan#98
* We do need dependencies to the AI providers, don't we?
* FixInAnYan#93
* Simplify code
* Partially fixInAnYan#92
* Fix checkstyle and localization
* Fix hyperlinks and text in ApiKeyMissingComponent
* Fixes from code review
* FixInAnYan#120
* Remove "X% work done" messages
* FixInAnYan#114
* Partially fixInAnYan#113
* Partially fixInAnYan#110
* FixInAnYan#110
* FixInAnYan#111
* Improve embedding model downloading notifications
* FixInAnYan#124
* FixInAnYan#122
* Fix wrong context window size when expert settings customization is turned off
* Attempt to fixInAnYan#95
* Finally fixInAnYan#105
* FixInAnYan#108
* Attempt to fixInAnYan#98
* Fix for InAnYan#104
* Fix for InAnYan#98
* Fix for InAnYan#95 (comment)
* Fix for InAnYan#98 (comment)
* Fix for InAnYan#126
* Fix for InAnYan#115
* Fix for InAnYan#113
* Fix for InAnYan#91
* Fix for InAnYan#121
* Fix for InAnYan#112 and InAnYan#116
* Fix for InAnYan#125
* Fixes from commit comments
* Fix for InAnYan#115
* Fix for InAnYan#120
* Fix for InAnYan#132
* Fix for InAnYan#132
* Fix for InAnYan#104
* Fix for InAnYan#118
* Fix for InAnYan#114
* Fix for InAnYan#104
* Store error messages in chat history
* Make error be a ChatMessageComponent
* Implement delete messages InAnYan#136
* Fix for InAnYan#118
* Fix for InAnYan#92
* Fix checkstyle and localization. And refactoring
* Fix for InAnYan#92
* Fix for InAnYan#139
* Show "Delete message" button only when necessary
* Fix for InAnYan#83
* Update src/main/java/org/jabref/logic/ai/AiService.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/logic/ai/chathistory/BibDatabaseChatHistoryManager.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/logic/ai/AiService.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/gui/Base.css
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/gui/Base.css
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Fix from code review
* Partial fix for InAnYan#125
* Update colors for error message
* Fix for InAnYan#145 and InAnYan#142
* Make progress for embedding model download
* Fix checkstyle and localization
* Add workaround to get FileHistoryMenuTest running again
* Small fixes
* Revert "Small fixes"
This reverts commit 85382a1.
* Introduce AiApiKeyProvider
* Fix IDE setup instructions
* Do not load API keys on startup
* Rely on keystore encryption
* Prevent mulitple rebuilds when muliple preferences are updated
* Fix localization to be more provider independent
* Fix method names
* Add poor man's solution to notify of API key changes
* Reduce calls to key store (and fix key saving)
* Fix for InAnYan#148 and partially InAnYan#146
* Revert "Fix for InAnYan#148 and partially InAnYan#146"
This reverts commit 5fa3bb5.
* Fix for scrolling down when deleting a message
* Sort EmbeddingModel enum variants
* Fix GenerateSummaryTask progress indication
* Fix dark mode
* Add notice for embedding models size
---------
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels

[outdated] type: feature

Projects


None yet


Milestone

v3.0

Development

Successfully merging this pull request may close these issues.

6 participants

@stefan-kolb@mlep@simonharrer@koppor@lenhard@lecc0r

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Full text crawlers by stefan-kolb · Pull Request #101 · JabRef/jabref · GitHub
Skip to content
Full text crawlers - #101
Merged
stefan-kolb merged 12 commits into
masterfrom 
catalog-crawler
Oct 9, 2015
Merged
Full text crawlers#101
stefan-kolb merged 12 commits into
masterfrom 
catalog-crawler
Conversation
@stefan-kolb
@stefan-kolbstefan-kolb
commented
Aug 14, 2015

Copy link


Copy Markdown

Member
This PR enables automatic PDF fulltext downloads.
Current catalogs:
  • ACS
  • arXiv
  • Springer
  • Sciencedirect (Elsevier)
  • Google Scholar
Questions:
  • Do we want to include more Crawlers? If yes, which?
  • Action was in the Tools menu. Imho it should be either auto downloaded or included with the downloador auto button in the detailed entry view.
TODO:
  •  Make it nonblocking and keep the progress bar!
  •  Add an option to delete the local file if a wrong file was downloaded
  •  Tests on Travis & CircleCI are blocked by Google by 403 Forbidden (Bot detection)
@stefan-kolbstefan-kolb
changed the title
Science catalog full text crawlers[WIP] Science catalog full text crawlersAug 14, 2015
@stefan-kolbstefan-kolb
changed the title
[WIP] Science catalog full text crawlers[WIP] Full text crawlersAug 14, 2015
@mlep
mlep
commented
Aug 17, 2015

Copy link


Copy Markdown

Contributor
About "Do we want to include more Crawlers? If yes, which?":

You have a gold mine in the section "Search (and import) from my specific database, please." in the sorted list of feature requests (this morning email).
@stefan-kolb
stefan-kolb
commented
Aug 17, 2015

Copy link


Copy Markdown

MemberAuthor
@mlep Thanks 👍
@simonharrer
simonharrer
commented
Aug 18, 2015

Copy link


Copy Markdown

Contributor
Your feature works pretty well. But there are cases in which a file can be downloaded but it is not the file that is intended. Thus, there must be a way to preview the pdf before adding it.
@stefan-kolb
stefan-kolb
commented
Aug 18, 2015

Copy link


Copy Markdown

MemberAuthor
We need to evaluate how often this will happen. Can you give me an example?
I think with a reasonable amount of crawlers this will work with a very high accuracy.

I would rather have the user open and check the file and then delete it and manually download it if it is not the correct file, as this will happen very seldom. If we use your approach we need to preview every PDF that will be anyway in > 90% of the times the correct one.
@simonharrer
simonharrer
commented
Aug 18, 2015

Copy link


Copy Markdown

Contributor
Ok, deletion after download is OK when it is done easily. At the moment, I cannot easily delete the file entry from the bibtex entry AND also delete the file from the hard drive. When we fix this, we do not need the check feature in the first place. Good idea!
My example was: I wanted to download the conference article Barros, Service Interaction Patterns, but got the techrep instead. Another issue was that I got a PDF of slides in some language I do not understand, probably czech when attempting to download a very old version of the BPMN standard.
@stefan-kolbstefan-kolb
mentioned this pull request
Aug 18, 2015

Closed

@stefan-kolb
stefan-kolbforce-pushed
the
catalog-crawler
branch
2 times, most recently
from
43d0c7d to
3384c4cCompareAugust 18, 2015 12:49
@koppor
koppor
commented
Aug 18, 2015

Copy link


Copy Markdown

Member
Regarding, "clean all used fields from Latex stuff, e.g. {~ etc.", this should be one of the "Edit -> Cleanup entries" functionality, isn't it? If not, this functionality should be added there, too.
Regarding "check for duplicates", JabRef offers "Search -> find duplicates". This functionality should be reused here.
@koppor
koppor
commented
Aug 18, 2015

Copy link


Copy Markdown

Member
A similar approach might have been taken by Christoph Lehner: https://sourceforge.net/p/jabref/discussion/318824/thread/6e5fea64/ Is it possible to synchronize somehow?

I think, this makes https://github.com/wbrenna/LocalCopy obsolete, doesn't it?
@stefan-kolb
stefan-kolb
commented
Aug 19, 2015

Copy link


Copy Markdown

MemberAuthor
The approach of C.Lehner is the base repository https://github.com/lehner/LocalCopy.

This PR is a native replacement for this and all forks.

Also, we don't support plugins anymore.
@stefan-kolb
stefan-kolbforce-pushed
the
catalog-crawler
branch
5 times, most recently
from
4399958 to
5b6c7a4CompareAugust 24, 2015 15:51
@stefan-kolbstefan-kolb
added this to the v2.80 milestone
Aug 27, 2015
@minkkinenminkkinen
mentioned this pull request
Sep 18, 2015

Closed

@stefan-kolbstefan-kolb
mentioned this pull request
Sep 28, 2015
 Merged


This was referenced Oct 7, 2015

Closed


Open

@lenhardlenhard
mentioned this pull request
Oct 9, 2015

Closed

@lenhard
lenhard
commented
Oct 9, 2015

Copy link


Copy Markdown

Member
What is missing to get this PR functional and integrated into master?
@lenhardlenhard
assigned lenhard and simonharrer and unassigned lenhardOct 9, 2015
@simonharrer
simonharrer
commented
Oct 9, 2015

Copy link


Copy Markdown

Contributor
This is just too messy to understand right now. Fixing this would require fixing the separation of GUI Event Thread with other Code parts - a major effort.
The issue is how to implement swing actions that require multiple user interactions during their task taking place. Normally, this would require a nesting of swing worker classes, one for each step and starting the next step within the EDT update method.
What is more, sometimes SwingWorker, and sometimes the spin Framework is used. All of this makes this issue even more complicated.
Someone else may take a look if they see this issue more clearly than me.
@simonharrersimonharrer
removed their assignment
Oct 9, 2015
@lenhard
lenhard
commented
Oct 9, 2015

Copy link


Copy Markdown

Member
So you are saying that you will not complete the PR? I cannot see anybody else who will. There is no point of having this hanging around in limbo until it deprecates. So we can close this PR without merging and close related open issues as won't fix.
@simonharrer: Please confirm!
@simonharrer
simonharrer
commented
Oct 9, 2015

Copy link


Copy Markdown

Contributor
Merge as good enough.
@stefan-kolbstefan-kolb
changed the title
[WIP] Full text crawlersFull text crawlersOct 9, 2015
stefan-kolb added a commit
that referenced
this pull request
Oct 9, 2015
@stefan-kolb
Merge pull request#101from JabRef/catalog-crawler
18a6a0e
Full text crawlers
@stefan-kolb
stefan-kolb
merged commit 18a6a0e
into
masterOct 9, 2015
@stefan-kolb
stefan-kolb
deleted the

catalog-crawler

branch
October 9, 2015 13:34
@kopporkoppor
mentioned this pull request
Oct 10, 2015

Closed

@lecc0r
lecc0r
commented
Dec 1, 2015

Copy link


Copy Markdown

Hi, I've just tested the newly implemented "Full text article download" function, however for several articles it downloaded a version from ResearchGate but I want to have the one from Elsevier. Dependent who you are working for, you have special access privileges on certain publisher sites and it would be great to specify in the preferences which crawler the download function should prefer. Perhaps even define a priority list.
@stefan-kolb
stefan-kolb
commented
Dec 1, 2015

Copy link


Copy Markdown

MemberAuthor
There is a hard-coded priority list right know which prefers the official publishers over google scholar for example. If you want to file a new feature request or enhancement please create a separate issue.
@koppor
koppor
commented
Dec 1, 2015

Copy link


Copy Markdown

Member
Done at #435.
InAnYan added a commit
that referenced
this pull request
Aug 3, 2024
@InAnYan
Fix#99and partially#101
c245e73
github-merge-queueBot
pushed a commit
that referenced
this pull request
Aug 14, 2024
@InAnYan@koppor
AI chatting functionality (#11430)
8a0edc2
* Fix the code from code review
* Fix from code review and create new AiChatTabWorking
* Improve chat history storage code
* More fix from code review
* Remove obsolete parameter
* Add JavaDoc comment
* Fix checkstyle
* Fix JavaDoc
* Fix more checkstyle
* More checkstyle fixes
* Fix code changes
* Improve the PR
* Rework ADR-0031 to enable to use another option
* Add many LOGGEr.trace statements
* Change "message window" to "context window"
* Fix compiler errors
* Fix issue list index issue of langchain4j
* Fix lint issue
* Update 0031-store-chats-alongside-database.md
* More tracing
* Refine logging
* Remove closing of AiChatLanguageModel (because it's not closable)
* Use external package for OpenAI API connection
* Provide a custom executor for RetrievalAugmentor
* Fix shutdown issue (I hope)
* Refactor classes
* Change BibDatabaseChatHistoryFile
* Revert BibDatabaseChatHistoryFile to old version because of langchain4j
* Make round corners for chat messages
* Refactor embeddings generation
* Refactor embeddings generation
* Refactor embeddings generation
* Fix CHANGELOG.md
* Remove jpro-mdfx
* Add comment
* Fix localizations
* Fix checkstyle and remove OpenAI from PRIVACY.md
* Remove unnecessary comments
* Fix privacy notice UI
* Introduce new ApiKeyMissingComponent
* Thanks Tobiaz Diez for writing such a good EntryEditorTab class
* Fix InAnYan/jabref issues
* Merge `build.gradle` and `settings.gradle` from main branch
* Update ADRs
* Implement rethought ADR for chat history
* Use OpenAI embedding model
* Use Deep Java embedding model
* Remove old langchain4j embedding models
* Fix checkstyle errors
* Fix checkstyle and remove old dependencies
* Fixes from code review
* Restructure
* Fix checkstyle errors
* Add API base URL parameter
* Fix localization
* Fix from code review + ADR
* Something broken
* Now MistralAI and Hugging Face work
* Fix base URL for other LLM providers
* Fix base URL for other LLM providers
* Refactor MVStore usage
* Load embedding model in background
* Bump langchain4j version
* Fix bug
* Fix checkstyle and localization
* Implement summarization
* Fix checkstyle and localization
* Improve PrivacyNoticeComponent
* Fix from code review
* Update localization
* Wrap text
* Add padding
* Fix markdown
* Use stuff algorithm
* Add GPT-4o-mini
* Make chat model editable
* Update context window size and summarization
* Fix checkstyle
* Update PrivacyNoticeComponent.fxml
* Update AI summary tab
* Fix localization
* Change order so that there is no diff
* Reorrder dependencies
* Add missing CHANGELOG.md entry
* Refine ADR-0033
* Refine ADR0034
* Fix typos
* Refine ADR-0036
* Fix ADR-0037
* Fix title case
* Fix changes in module-info.java
* Readd removed requires org.apache.httpcomponents.core5.httpcore5
* Revert change in JabRefGUI to avoid conflicts
* Remove empty lines
* Reorder entries in JabRef_en.properties
* Simplify SummariesStorage (and add test)
* Use region/endregion
* Fix position of comment
* Add comment why the event bus is needed
* Do not show exception to the user - just that an error is occurred (saves %0 in localization)
* Use "URL %0" without colon (consistency)
* Fix typos
* History has to be kept
* Remove empty lines
* Fix language (hopefully)
* Compilefix
* Simplify BibDatabaseChatHistoryManager
* Fix from code review
* Fix issue #103
* Rework embeddings cache clearing
* Fix#99 and partially #101
* Partially fixing shutdown issues and UI progress monitor issue
* Add "requires scala.library" and add "region:" / "endregion"
* More grouping (move de.saxsys.mvvmfx.validation up)
* Add alphabetical hint
* FixInAnYan#101 and InAnYan#106
* Discard changes to settings.gradle
* FixInAnYan#105
* Follow-up fix for InAnYan#103
* Follow-up fix for InAnYan#103
* Remove obsolete class
* Partially fixInAnYan#98
* We do need dependencies to the AI providers, don't we?
* FixInAnYan#93
* Simplify code
* Partially fixInAnYan#92
* Fix checkstyle and localization
* Fix hyperlinks and text in ApiKeyMissingComponent
* Fixes from code review
* FixInAnYan#120
* Remove "X% work done" messages
* FixInAnYan#114
* Partially fixInAnYan#113
* Partially fixInAnYan#110
* FixInAnYan#110
* FixInAnYan#111
* Improve embedding model downloading notifications
* FixInAnYan#124
* FixInAnYan#122
* Fix wrong context window size when expert settings customization is turned off
* Attempt to fixInAnYan#95
* Finally fixInAnYan#105
* FixInAnYan#108
* Attempt to fixInAnYan#98
* Fix for InAnYan#104
* Fix for InAnYan#98
* Fix for InAnYan#95 (comment)
* Fix for InAnYan#98 (comment)
* Fix for InAnYan#126
* Fix for InAnYan#115
* Fix for InAnYan#113
* Fix for InAnYan#91
* Fix for InAnYan#121
* Fix for InAnYan#112 and InAnYan#116
* Fix for InAnYan#125
* Fixes from commit comments
* Fix for InAnYan#115
* Fix for InAnYan#120
* Fix for InAnYan#132
* Fix for InAnYan#132
* Fix for InAnYan#104
* Fix for InAnYan#118
* Fix for InAnYan#114
* Fix for InAnYan#104
* Store error messages in chat history
* Make error be a ChatMessageComponent
* Implement delete messages InAnYan#136
* Fix for InAnYan#118
* Fix for InAnYan#92
* Fix checkstyle and localization. And refactoring
* Fix for InAnYan#92
* Fix for InAnYan#139
* Show "Delete message" button only when necessary
* Fix for InAnYan#83
* Update src/main/java/org/jabref/logic/ai/AiService.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/logic/ai/chathistory/BibDatabaseChatHistoryManager.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/logic/ai/AiService.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/gui/Base.css
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/gui/Base.css
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Fix from code review
* Partial fix for InAnYan#125
* Update colors for error message
* Fix for InAnYan#145 and InAnYan#142
* Make progress for embedding model download
* Fix checkstyle and localization
* Add workaround to get FileHistoryMenuTest running again
* Small fixes
* Revert "Small fixes"
This reverts commit 85382a1.
* Introduce AiApiKeyProvider
* Fix IDE setup instructions
* Do not load API keys on startup
* Rely on keystore encryption
* Prevent mulitple rebuilds when muliple preferences are updated
* Fix localization to be more provider independent
* Fix method names
* Add poor man's solution to notify of API key changes
* Reduce calls to key store (and fix key saving)
* Fix for InAnYan#148 and partially InAnYan#146
* Revert "Fix for InAnYan#148 and partially InAnYan#146"
This reverts commit 5fa3bb5.
* Fix for scrolling down when deleting a message
* Sort EmbeddingModel enum variants
* Fix GenerateSummaryTask progress indication
* Fix dark mode
* Add notice for embedding models size
---------
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels

[outdated] type: feature

Projects


None yet


Milestone

v3.0

Development

Successfully merging this pull request may close these issues.

6 participants

@stefan-kolb@mlep@simonharrer@koppor@lenhard@lecc0r

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Full text crawlers by stefan-kolb · Pull Request #101 · JabRef/jabref · GitHub
Skip to content
Full text crawlers - #101
Merged
stefan-kolb merged 12 commits into
masterfrom 
catalog-crawler
Oct 9, 2015
Merged
Full text crawlers#101
stefan-kolb merged 12 commits into
masterfrom 
catalog-crawler
Conversation
@stefan-kolb
@stefan-kolbstefan-kolb
commented
Aug 14, 2015

Copy link


Copy Markdown

Member
This PR enables automatic PDF fulltext downloads.
Current catalogs:
  • ACS
  • arXiv
  • Springer
  • Sciencedirect (Elsevier)
  • Google Scholar
Questions:
  • Do we want to include more Crawlers? If yes, which?
  • Action was in the Tools menu. Imho it should be either auto downloaded or included with the downloador auto button in the detailed entry view.
TODO:
  •  Make it nonblocking and keep the progress bar!
  •  Add an option to delete the local file if a wrong file was downloaded
  •  Tests on Travis & CircleCI are blocked by Google by 403 Forbidden (Bot detection)
@stefan-kolbstefan-kolb
changed the title
Science catalog full text crawlers[WIP] Science catalog full text crawlersAug 14, 2015
@stefan-kolbstefan-kolb
changed the title
[WIP] Science catalog full text crawlers[WIP] Full text crawlersAug 14, 2015
@mlep
mlep
commented
Aug 17, 2015

Copy link


Copy Markdown

Contributor
About "Do we want to include more Crawlers? If yes, which?":

You have a gold mine in the section "Search (and import) from my specific database, please." in the sorted list of feature requests (this morning email).
@stefan-kolb
stefan-kolb
commented
Aug 17, 2015

Copy link


Copy Markdown

MemberAuthor
@mlep Thanks 👍
@simonharrer
simonharrer
commented
Aug 18, 2015

Copy link


Copy Markdown

Contributor
Your feature works pretty well. But there are cases in which a file can be downloaded but it is not the file that is intended. Thus, there must be a way to preview the pdf before adding it.
@stefan-kolb
stefan-kolb
commented
Aug 18, 2015

Copy link


Copy Markdown

MemberAuthor
We need to evaluate how often this will happen. Can you give me an example?
I think with a reasonable amount of crawlers this will work with a very high accuracy.

I would rather have the user open and check the file and then delete it and manually download it if it is not the correct file, as this will happen very seldom. If we use your approach we need to preview every PDF that will be anyway in > 90% of the times the correct one.
@simonharrer
simonharrer
commented
Aug 18, 2015

Copy link


Copy Markdown

Contributor
Ok, deletion after download is OK when it is done easily. At the moment, I cannot easily delete the file entry from the bibtex entry AND also delete the file from the hard drive. When we fix this, we do not need the check feature in the first place. Good idea!
My example was: I wanted to download the conference article Barros, Service Interaction Patterns, but got the techrep instead. Another issue was that I got a PDF of slides in some language I do not understand, probably czech when attempting to download a very old version of the BPMN standard.
@stefan-kolbstefan-kolb
mentioned this pull request
Aug 18, 2015

Closed

@stefan-kolb
stefan-kolbforce-pushed
the
catalog-crawler
branch
2 times, most recently
from
43d0c7d to
3384c4cCompareAugust 18, 2015 12:49
@koppor
koppor
commented
Aug 18, 2015

Copy link


Copy Markdown

Member
Regarding, "clean all used fields from Latex stuff, e.g. {~ etc.", this should be one of the "Edit -> Cleanup entries" functionality, isn't it? If not, this functionality should be added there, too.
Regarding "check for duplicates", JabRef offers "Search -> find duplicates". This functionality should be reused here.
@koppor
koppor
commented
Aug 18, 2015

Copy link


Copy Markdown

Member
A similar approach might have been taken by Christoph Lehner: https://sourceforge.net/p/jabref/discussion/318824/thread/6e5fea64/ Is it possible to synchronize somehow?

I think, this makes https://github.com/wbrenna/LocalCopy obsolete, doesn't it?
@stefan-kolb
stefan-kolb
commented
Aug 19, 2015

Copy link


Copy Markdown

MemberAuthor
The approach of C.Lehner is the base repository https://github.com/lehner/LocalCopy.

This PR is a native replacement for this and all forks.

Also, we don't support plugins anymore.
@stefan-kolb
stefan-kolbforce-pushed
the
catalog-crawler
branch
5 times, most recently
from
4399958 to
5b6c7a4CompareAugust 24, 2015 15:51
@stefan-kolbstefan-kolb
added this to the v2.80 milestone
Aug 27, 2015
@minkkinenminkkinen
mentioned this pull request
Sep 18, 2015

Closed

@stefan-kolbstefan-kolb
mentioned this pull request
Sep 28, 2015
 Merged


This was referenced Oct 7, 2015

Closed


Open

@lenhardlenhard
mentioned this pull request
Oct 9, 2015

Closed

@lenhard
lenhard
commented
Oct 9, 2015

Copy link


Copy Markdown

Member
What is missing to get this PR functional and integrated into master?
@lenhardlenhard
assigned lenhard and simonharrer and unassigned lenhardOct 9, 2015
@simonharrer
simonharrer
commented
Oct 9, 2015

Copy link


Copy Markdown

Contributor
This is just too messy to understand right now. Fixing this would require fixing the separation of GUI Event Thread with other Code parts - a major effort.
The issue is how to implement swing actions that require multiple user interactions during their task taking place. Normally, this would require a nesting of swing worker classes, one for each step and starting the next step within the EDT update method.
What is more, sometimes SwingWorker, and sometimes the spin Framework is used. All of this makes this issue even more complicated.
Someone else may take a look if they see this issue more clearly than me.
@simonharrersimonharrer
removed their assignment
Oct 9, 2015
@lenhard
lenhard
commented
Oct 9, 2015

Copy link


Copy Markdown

Member
So you are saying that you will not complete the PR? I cannot see anybody else who will. There is no point of having this hanging around in limbo until it deprecates. So we can close this PR without merging and close related open issues as won't fix.
@simonharrer: Please confirm!
@simonharrer
simonharrer
commented
Oct 9, 2015

Copy link


Copy Markdown

Contributor
Merge as good enough.
@stefan-kolbstefan-kolb
changed the title
[WIP] Full text crawlersFull text crawlersOct 9, 2015
stefan-kolb added a commit
that referenced
this pull request
Oct 9, 2015
@stefan-kolb
Merge pull request#101from JabRef/catalog-crawler
18a6a0e
Full text crawlers
@stefan-kolb
stefan-kolb
merged commit 18a6a0e
into
masterOct 9, 2015
@stefan-kolb
stefan-kolb
deleted the

catalog-crawler

branch
October 9, 2015 13:34
@kopporkoppor
mentioned this pull request
Oct 10, 2015

Closed

@lecc0r
lecc0r
commented
Dec 1, 2015

Copy link


Copy Markdown

Hi, I've just tested the newly implemented "Full text article download" function, however for several articles it downloaded a version from ResearchGate but I want to have the one from Elsevier. Dependent who you are working for, you have special access privileges on certain publisher sites and it would be great to specify in the preferences which crawler the download function should prefer. Perhaps even define a priority list.
@stefan-kolb
stefan-kolb
commented
Dec 1, 2015

Copy link


Copy Markdown

MemberAuthor
There is a hard-coded priority list right know which prefers the official publishers over google scholar for example. If you want to file a new feature request or enhancement please create a separate issue.
@koppor
koppor
commented
Dec 1, 2015

Copy link


Copy Markdown

Member
Done at #435.
InAnYan added a commit
that referenced
this pull request
Aug 3, 2024
@InAnYan
Fix#99and partially#101
c245e73
github-merge-queueBot
pushed a commit
that referenced
this pull request
Aug 14, 2024
@InAnYan@koppor
AI chatting functionality (#11430)
8a0edc2
* Fix the code from code review
* Fix from code review and create new AiChatTabWorking
* Improve chat history storage code
* More fix from code review
* Remove obsolete parameter
* Add JavaDoc comment
* Fix checkstyle
* Fix JavaDoc
* Fix more checkstyle
* More checkstyle fixes
* Fix code changes
* Improve the PR
* Rework ADR-0031 to enable to use another option
* Add many LOGGEr.trace statements
* Change "message window" to "context window"
* Fix compiler errors
* Fix issue list index issue of langchain4j
* Fix lint issue
* Update 0031-store-chats-alongside-database.md
* More tracing
* Refine logging
* Remove closing of AiChatLanguageModel (because it's not closable)
* Use external package for OpenAI API connection
* Provide a custom executor for RetrievalAugmentor
* Fix shutdown issue (I hope)
* Refactor classes
* Change BibDatabaseChatHistoryFile
* Revert BibDatabaseChatHistoryFile to old version because of langchain4j
* Make round corners for chat messages
* Refactor embeddings generation
* Refactor embeddings generation
* Refactor embeddings generation
* Fix CHANGELOG.md
* Remove jpro-mdfx
* Add comment
* Fix localizations
* Fix checkstyle and remove OpenAI from PRIVACY.md
* Remove unnecessary comments
* Fix privacy notice UI
* Introduce new ApiKeyMissingComponent
* Thanks Tobiaz Diez for writing such a good EntryEditorTab class
* Fix InAnYan/jabref issues
* Merge `build.gradle` and `settings.gradle` from main branch
* Update ADRs
* Implement rethought ADR for chat history
* Use OpenAI embedding model
* Use Deep Java embedding model
* Remove old langchain4j embedding models
* Fix checkstyle errors
* Fix checkstyle and remove old dependencies
* Fixes from code review
* Restructure
* Fix checkstyle errors
* Add API base URL parameter
* Fix localization
* Fix from code review + ADR
* Something broken
* Now MistralAI and Hugging Face work
* Fix base URL for other LLM providers
* Fix base URL for other LLM providers
* Refactor MVStore usage
* Load embedding model in background
* Bump langchain4j version
* Fix bug
* Fix checkstyle and localization
* Implement summarization
* Fix checkstyle and localization
* Improve PrivacyNoticeComponent
* Fix from code review
* Update localization
* Wrap text
* Add padding
* Fix markdown
* Use stuff algorithm
* Add GPT-4o-mini
* Make chat model editable
* Update context window size and summarization
* Fix checkstyle
* Update PrivacyNoticeComponent.fxml
* Update AI summary tab
* Fix localization
* Change order so that there is no diff
* Reorrder dependencies
* Add missing CHANGELOG.md entry
* Refine ADR-0033
* Refine ADR0034
* Fix typos
* Refine ADR-0036
* Fix ADR-0037
* Fix title case
* Fix changes in module-info.java
* Readd removed requires org.apache.httpcomponents.core5.httpcore5
* Revert change in JabRefGUI to avoid conflicts
* Remove empty lines
* Reorder entries in JabRef_en.properties
* Simplify SummariesStorage (and add test)
* Use region/endregion
* Fix position of comment
* Add comment why the event bus is needed
* Do not show exception to the user - just that an error is occurred (saves %0 in localization)
* Use "URL %0" without colon (consistency)
* Fix typos
* History has to be kept
* Remove empty lines
* Fix language (hopefully)
* Compilefix
* Simplify BibDatabaseChatHistoryManager
* Fix from code review
* Fix issue #103
* Rework embeddings cache clearing
* Fix#99 and partially #101
* Partially fixing shutdown issues and UI progress monitor issue
* Add "requires scala.library" and add "region:" / "endregion"
* More grouping (move de.saxsys.mvvmfx.validation up)
* Add alphabetical hint
* FixInAnYan#101 and InAnYan#106
* Discard changes to settings.gradle
* FixInAnYan#105
* Follow-up fix for InAnYan#103
* Follow-up fix for InAnYan#103
* Remove obsolete class
* Partially fixInAnYan#98
* We do need dependencies to the AI providers, don't we?
* FixInAnYan#93
* Simplify code
* Partially fixInAnYan#92
* Fix checkstyle and localization
* Fix hyperlinks and text in ApiKeyMissingComponent
* Fixes from code review
* FixInAnYan#120
* Remove "X% work done" messages
* FixInAnYan#114
* Partially fixInAnYan#113
* Partially fixInAnYan#110
* FixInAnYan#110
* FixInAnYan#111
* Improve embedding model downloading notifications
* FixInAnYan#124
* FixInAnYan#122
* Fix wrong context window size when expert settings customization is turned off
* Attempt to fixInAnYan#95
* Finally fixInAnYan#105
* FixInAnYan#108
* Attempt to fixInAnYan#98
* Fix for InAnYan#104
* Fix for InAnYan#98
* Fix for InAnYan#95 (comment)
* Fix for InAnYan#98 (comment)
* Fix for InAnYan#126
* Fix for InAnYan#115
* Fix for InAnYan#113
* Fix for InAnYan#91
* Fix for InAnYan#121
* Fix for InAnYan#112 and InAnYan#116
* Fix for InAnYan#125
* Fixes from commit comments
* Fix for InAnYan#115
* Fix for InAnYan#120
* Fix for InAnYan#132
* Fix for InAnYan#132
* Fix for InAnYan#104
* Fix for InAnYan#118
* Fix for InAnYan#114
* Fix for InAnYan#104
* Store error messages in chat history
* Make error be a ChatMessageComponent
* Implement delete messages InAnYan#136
* Fix for InAnYan#118
* Fix for InAnYan#92
* Fix checkstyle and localization. And refactoring
* Fix for InAnYan#92
* Fix for InAnYan#139
* Show "Delete message" button only when necessary
* Fix for InAnYan#83
* Update src/main/java/org/jabref/logic/ai/AiService.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/logic/ai/chathistory/BibDatabaseChatHistoryManager.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/logic/ai/AiService.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/gui/Base.css
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/gui/Base.css
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Fix from code review
* Partial fix for InAnYan#125
* Update colors for error message
* Fix for InAnYan#145 and InAnYan#142
* Make progress for embedding model download
* Fix checkstyle and localization
* Add workaround to get FileHistoryMenuTest running again
* Small fixes
* Revert "Small fixes"
This reverts commit 85382a1.
* Introduce AiApiKeyProvider
* Fix IDE setup instructions
* Do not load API keys on startup
* Rely on keystore encryption
* Prevent mulitple rebuilds when muliple preferences are updated
* Fix localization to be more provider independent
* Fix method names
* Add poor man's solution to notify of API key changes
* Reduce calls to key store (and fix key saving)
* Fix for InAnYan#148 and partially InAnYan#146
* Revert "Fix for InAnYan#148 and partially InAnYan#146"
This reverts commit 5fa3bb5.
* Fix for scrolling down when deleting a message
* Sort EmbeddingModel enum variants
* Fix GenerateSummaryTask progress indication
* Fix dark mode
* Add notice for embedding models size
---------
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels

[outdated] type: feature

Projects


None yet


Milestone

v3.0

Development

Successfully merging this pull request may close these issues.

6 participants

@stefan-kolb@mlep@simonharrer@koppor@lenhard@lecc0r

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
Full text crawlers by stefan-kolb · Pull Request #101 · JabRef/jabref · GitHub
Skip to content
Full text crawlers - #101
Merged
stefan-kolb merged 12 commits into
masterfrom 
catalog-crawler
Oct 9, 2015
Merged
Full text crawlers#101
stefan-kolb merged 12 commits into
masterfrom 
catalog-crawler
Conversation
@stefan-kolb
@stefan-kolbstefan-kolb
commented
Aug 14, 2015

Copy link


Copy Markdown

Member
This PR enables automatic PDF fulltext downloads.
Current catalogs:
  • ACS
  • arXiv
  • Springer
  • Sciencedirect (Elsevier)
  • Google Scholar
Questions:
  • Do we want to include more Crawlers? If yes, which?
  • Action was in the Tools menu. Imho it should be either auto downloaded or included with the downloador auto button in the detailed entry view.
TODO:
  •  Make it nonblocking and keep the progress bar!
  •  Add an option to delete the local file if a wrong file was downloaded
  •  Tests on Travis & CircleCI are blocked by Google by 403 Forbidden (Bot detection)
@stefan-kolbstefan-kolb
changed the title
Science catalog full text crawlers[WIP] Science catalog full text crawlersAug 14, 2015
@stefan-kolbstefan-kolb
changed the title
[WIP] Science catalog full text crawlers[WIP] Full text crawlersAug 14, 2015
@mlep
mlep
commented
Aug 17, 2015

Copy link


Copy Markdown

Contributor
About "Do we want to include more Crawlers? If yes, which?":

You have a gold mine in the section "Search (and import) from my specific database, please." in the sorted list of feature requests (this morning email).
@stefan-kolb
stefan-kolb
commented
Aug 17, 2015

Copy link


Copy Markdown

MemberAuthor
@mlep Thanks 👍
@simonharrer
simonharrer
commented
Aug 18, 2015

Copy link


Copy Markdown

Contributor
Your feature works pretty well. But there are cases in which a file can be downloaded but it is not the file that is intended. Thus, there must be a way to preview the pdf before adding it.
@stefan-kolb
stefan-kolb
commented
Aug 18, 2015

Copy link


Copy Markdown

MemberAuthor
We need to evaluate how often this will happen. Can you give me an example?
I think with a reasonable amount of crawlers this will work with a very high accuracy.

I would rather have the user open and check the file and then delete it and manually download it if it is not the correct file, as this will happen very seldom. If we use your approach we need to preview every PDF that will be anyway in > 90% of the times the correct one.
@simonharrer
simonharrer
commented
Aug 18, 2015

Copy link


Copy Markdown

Contributor
Ok, deletion after download is OK when it is done easily. At the moment, I cannot easily delete the file entry from the bibtex entry AND also delete the file from the hard drive. When we fix this, we do not need the check feature in the first place. Good idea!
My example was: I wanted to download the conference article Barros, Service Interaction Patterns, but got the techrep instead. Another issue was that I got a PDF of slides in some language I do not understand, probably czech when attempting to download a very old version of the BPMN standard.
@stefan-kolbstefan-kolb
mentioned this pull request
Aug 18, 2015

Closed

@stefan-kolb
stefan-kolbforce-pushed
the
catalog-crawler
branch
2 times, most recently
from
43d0c7d to
3384c4cCompareAugust 18, 2015 12:49
@koppor
koppor
commented
Aug 18, 2015

Copy link


Copy Markdown

Member
Regarding, "clean all used fields from Latex stuff, e.g. {~ etc.", this should be one of the "Edit -> Cleanup entries" functionality, isn't it? If not, this functionality should be added there, too.
Regarding "check for duplicates", JabRef offers "Search -> find duplicates". This functionality should be reused here.
@koppor
koppor
commented
Aug 18, 2015

Copy link


Copy Markdown

Member
A similar approach might have been taken by Christoph Lehner: https://sourceforge.net/p/jabref/discussion/318824/thread/6e5fea64/ Is it possible to synchronize somehow?

I think, this makes https://github.com/wbrenna/LocalCopy obsolete, doesn't it?
@stefan-kolb
stefan-kolb
commented
Aug 19, 2015

Copy link


Copy Markdown

MemberAuthor
The approach of C.Lehner is the base repository https://github.com/lehner/LocalCopy.

This PR is a native replacement for this and all forks.

Also, we don't support plugins anymore.
@stefan-kolb
stefan-kolbforce-pushed
the
catalog-crawler
branch
5 times, most recently
from
4399958 to
5b6c7a4CompareAugust 24, 2015 15:51
@stefan-kolbstefan-kolb
added this to the v2.80 milestone
Aug 27, 2015
@minkkinenminkkinen
mentioned this pull request
Sep 18, 2015

Closed

@stefan-kolbstefan-kolb
mentioned this pull request
Sep 28, 2015
 Merged


This was referenced Oct 7, 2015

Closed


Open

@lenhardlenhard
mentioned this pull request
Oct 9, 2015

Closed

@lenhard
lenhard
commented
Oct 9, 2015

Copy link


Copy Markdown

Member
What is missing to get this PR functional and integrated into master?
@lenhardlenhard
assigned lenhard and simonharrer and unassigned lenhardOct 9, 2015
@simonharrer
simonharrer
commented
Oct 9, 2015

Copy link


Copy Markdown

Contributor
This is just too messy to understand right now. Fixing this would require fixing the separation of GUI Event Thread with other Code parts - a major effort.
The issue is how to implement swing actions that require multiple user interactions during their task taking place. Normally, this would require a nesting of swing worker classes, one for each step and starting the next step within the EDT update method.
What is more, sometimes SwingWorker, and sometimes the spin Framework is used. All of this makes this issue even more complicated.
Someone else may take a look if they see this issue more clearly than me.
@simonharrersimonharrer
removed their assignment
Oct 9, 2015
@lenhard
lenhard
commented
Oct 9, 2015

Copy link


Copy Markdown

Member
So you are saying that you will not complete the PR? I cannot see anybody else who will. There is no point of having this hanging around in limbo until it deprecates. So we can close this PR without merging and close related open issues as won't fix.
@simonharrer: Please confirm!
@simonharrer
simonharrer
commented
Oct 9, 2015

Copy link


Copy Markdown

Contributor
Merge as good enough.
@stefan-kolbstefan-kolb
changed the title
[WIP] Full text crawlersFull text crawlersOct 9, 2015
stefan-kolb added a commit
that referenced
this pull request
Oct 9, 2015
@stefan-kolb
Merge pull request#101from JabRef/catalog-crawler
18a6a0e
Full text crawlers
@stefan-kolb
stefan-kolb
merged commit 18a6a0e
into
masterOct 9, 2015
@stefan-kolb
stefan-kolb
deleted the

catalog-crawler

branch
October 9, 2015 13:34
@kopporkoppor
mentioned this pull request
Oct 10, 2015

Closed

@lecc0r
lecc0r
commented
Dec 1, 2015

Copy link


Copy Markdown

Hi, I've just tested the newly implemented "Full text article download" function, however for several articles it downloaded a version from ResearchGate but I want to have the one from Elsevier. Dependent who you are working for, you have special access privileges on certain publisher sites and it would be great to specify in the preferences which crawler the download function should prefer. Perhaps even define a priority list.
@stefan-kolb
stefan-kolb
commented
Dec 1, 2015

Copy link


Copy Markdown

MemberAuthor
There is a hard-coded priority list right know which prefers the official publishers over google scholar for example. If you want to file a new feature request or enhancement please create a separate issue.
@koppor
koppor
commented
Dec 1, 2015

Copy link


Copy Markdown

Member
Done at #435.
InAnYan added a commit
that referenced
this pull request
Aug 3, 2024
@InAnYan
Fix#99and partially#101
c245e73
github-merge-queueBot
pushed a commit
that referenced
this pull request
Aug 14, 2024
@InAnYan@koppor
AI chatting functionality (#11430)
8a0edc2
* Fix the code from code review
* Fix from code review and create new AiChatTabWorking
* Improve chat history storage code
* More fix from code review
* Remove obsolete parameter
* Add JavaDoc comment
* Fix checkstyle
* Fix JavaDoc
* Fix more checkstyle
* More checkstyle fixes
* Fix code changes
* Improve the PR
* Rework ADR-0031 to enable to use another option
* Add many LOGGEr.trace statements
* Change "message window" to "context window"
* Fix compiler errors
* Fix issue list index issue of langchain4j
* Fix lint issue
* Update 0031-store-chats-alongside-database.md
* More tracing
* Refine logging
* Remove closing of AiChatLanguageModel (because it's not closable)
* Use external package for OpenAI API connection
* Provide a custom executor for RetrievalAugmentor
* Fix shutdown issue (I hope)
* Refactor classes
* Change BibDatabaseChatHistoryFile
* Revert BibDatabaseChatHistoryFile to old version because of langchain4j
* Make round corners for chat messages
* Refactor embeddings generation
* Refactor embeddings generation
* Refactor embeddings generation
* Fix CHANGELOG.md
* Remove jpro-mdfx
* Add comment
* Fix localizations
* Fix checkstyle and remove OpenAI from PRIVACY.md
* Remove unnecessary comments
* Fix privacy notice UI
* Introduce new ApiKeyMissingComponent
* Thanks Tobiaz Diez for writing such a good EntryEditorTab class
* Fix InAnYan/jabref issues
* Merge `build.gradle` and `settings.gradle` from main branch
* Update ADRs
* Implement rethought ADR for chat history
* Use OpenAI embedding model
* Use Deep Java embedding model
* Remove old langchain4j embedding models
* Fix checkstyle errors
* Fix checkstyle and remove old dependencies
* Fixes from code review
* Restructure
* Fix checkstyle errors
* Add API base URL parameter
* Fix localization
* Fix from code review + ADR
* Something broken
* Now MistralAI and Hugging Face work
* Fix base URL for other LLM providers
* Fix base URL for other LLM providers
* Refactor MVStore usage
* Load embedding model in background
* Bump langchain4j version
* Fix bug
* Fix checkstyle and localization
* Implement summarization
* Fix checkstyle and localization
* Improve PrivacyNoticeComponent
* Fix from code review
* Update localization
* Wrap text
* Add padding
* Fix markdown
* Use stuff algorithm
* Add GPT-4o-mini
* Make chat model editable
* Update context window size and summarization
* Fix checkstyle
* Update PrivacyNoticeComponent.fxml
* Update AI summary tab
* Fix localization
* Change order so that there is no diff
* Reorrder dependencies
* Add missing CHANGELOG.md entry
* Refine ADR-0033
* Refine ADR0034
* Fix typos
* Refine ADR-0036
* Fix ADR-0037
* Fix title case
* Fix changes in module-info.java
* Readd removed requires org.apache.httpcomponents.core5.httpcore5
* Revert change in JabRefGUI to avoid conflicts
* Remove empty lines
* Reorder entries in JabRef_en.properties
* Simplify SummariesStorage (and add test)
* Use region/endregion
* Fix position of comment
* Add comment why the event bus is needed
* Do not show exception to the user - just that an error is occurred (saves %0 in localization)
* Use "URL %0" without colon (consistency)
* Fix typos
* History has to be kept
* Remove empty lines
* Fix language (hopefully)
* Compilefix
* Simplify BibDatabaseChatHistoryManager
* Fix from code review
* Fix issue #103
* Rework embeddings cache clearing
* Fix#99 and partially #101
* Partially fixing shutdown issues and UI progress monitor issue
* Add "requires scala.library" and add "region:" / "endregion"
* More grouping (move de.saxsys.mvvmfx.validation up)
* Add alphabetical hint
* FixInAnYan#101 and InAnYan#106
* Discard changes to settings.gradle
* FixInAnYan#105
* Follow-up fix for InAnYan#103
* Follow-up fix for InAnYan#103
* Remove obsolete class
* Partially fixInAnYan#98
* We do need dependencies to the AI providers, don't we?
* FixInAnYan#93
* Simplify code
* Partially fixInAnYan#92
* Fix checkstyle and localization
* Fix hyperlinks and text in ApiKeyMissingComponent
* Fixes from code review
* FixInAnYan#120
* Remove "X% work done" messages
* FixInAnYan#114
* Partially fixInAnYan#113
* Partially fixInAnYan#110
* FixInAnYan#110
* FixInAnYan#111
* Improve embedding model downloading notifications
* FixInAnYan#124
* FixInAnYan#122
* Fix wrong context window size when expert settings customization is turned off
* Attempt to fixInAnYan#95
* Finally fixInAnYan#105
* FixInAnYan#108
* Attempt to fixInAnYan#98
* Fix for InAnYan#104
* Fix for InAnYan#98
* Fix for InAnYan#95 (comment)
* Fix for InAnYan#98 (comment)
* Fix for InAnYan#126
* Fix for InAnYan#115
* Fix for InAnYan#113
* Fix for InAnYan#91
* Fix for InAnYan#121
* Fix for InAnYan#112 and InAnYan#116
* Fix for InAnYan#125
* Fixes from commit comments
* Fix for InAnYan#115
* Fix for InAnYan#120
* Fix for InAnYan#132
* Fix for InAnYan#132
* Fix for InAnYan#104
* Fix for InAnYan#118
* Fix for InAnYan#114
* Fix for InAnYan#104
* Store error messages in chat history
* Make error be a ChatMessageComponent
* Implement delete messages InAnYan#136
* Fix for InAnYan#118
* Fix for InAnYan#92
* Fix checkstyle and localization. And refactoring
* Fix for InAnYan#92
* Fix for InAnYan#139
* Show "Delete message" button only when necessary
* Fix for InAnYan#83
* Update src/main/java/org/jabref/logic/ai/AiService.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/logic/ai/chathistory/BibDatabaseChatHistoryManager.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/logic/ai/AiService.java
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/gui/Base.css
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Update src/main/java/org/jabref/gui/Base.css
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
* Fix from code review
* Partial fix for InAnYan#125
* Update colors for error message
* Fix for InAnYan#145 and InAnYan#142
* Make progress for embedding model download
* Fix checkstyle and localization
* Add workaround to get FileHistoryMenuTest running again
* Small fixes
* Revert "Small fixes"
This reverts commit 85382a1.
* Introduce AiApiKeyProvider
* Fix IDE setup instructions
* Do not load API keys on startup
* Rely on keystore encryption
* Prevent mulitple rebuilds when muliple preferences are updated
* Fix localization to be more provider independent
* Fix method names
* Add poor man's solution to notify of API key changes
* Reduce calls to key store (and fix key saving)
* Fix for InAnYan#148 and partially InAnYan#146
* Revert "Fix for InAnYan#148 and partially InAnYan#146"
This reverts commit 5fa3bb5.
* Fix for scrolling down when deleting a message
* Sort EmbeddingModel enum variants
* Fix GenerateSummaryTask progress indication
* Fix dark mode
* Add notice for embedding models size
---------
Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels

[outdated] type: feature

Projects


None yet


Milestone

v3.0

Development

Successfully merging this pull request may close these issues.

6 participants

@stefan-kolb@mlep@simonharrer@koppor@lenhard@lecc0r