Uh oh!
There was an error while loading. Please reload this page.
HBASE-26192 Master UI hbck should provide a JSON formatted output option - #4470
Closed
apurtell wants to merge 2 commits into
Closed
HBASE-26192 Master UI hbck should provide a JSON formatted output option#4470apurtell wants to merge 2 commits into
apurtell wants to merge 2 commits into
Conversation
apurtell
commented
May 27, 2022
ContributorAuthor
Needs some basic tests; up next. |
Apache-HBase
commented
May 27, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
May 27, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
May 27, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
May 27, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
May 27, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
May 27, 2022
🎊 +1 overall
This message was automatically generated. |
apurtell
marked this pull request as draft
May 27, 2022 21:35
apurtell
commented
May 27, 2022
ContributorAuthor
Rebase. Test implementation in progress. |
Apache-HBase
commented
May 27, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
May 28, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
May 28, 2022
💔 -1 overall
This message was automatically generated. |
| private static final long serialVersionUID = 1L; | ||
| private static final Logger LOG = LoggerFactory.getLogger(MasterHbckServlet.class); | ||
| private static final Gson GSON = new Gson(); |
Member
There was a problem hiding this comment.
You almost certainly don't want to use the Gson object defaults; I would expect some minor configuration. See also hbase-server/src/main/java/.../hbase/master/http/gson/GsonFactory.java
Contributor
There was a problem hiding this comment.
GsonUtil would also work but yes GsonFactory#buildGson should be quite enough.
apurtell
commented
May 31, 2022
ContributorAuthor
Rebased, with test |
apurtell
commented
May 31, 2022
ContributorAuthor
Clarify something in test and apply spotless. |
Apache-HBase
commented
May 31, 2022
🎊 +1 overall
This message was automatically generated. |
apurtell
commented
Jun 1, 2022
ContributorAuthor
Example - healthy cluster:
|
Apache-HBase
commented
Jun 1, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 1, 2022
🎊 +1 overall
This message was automatically generated. |
ContributorAuthor
@ndimiduk Do you have any concerns with the current state of this? I think if someone wants to use this servlet to check status with a json aware tool it will provide all the necessary information, but we don't have an API to recover the original Java types from the GSON serialization. OTOH I do not think that is required. @virajjasani WDYT? |
virajjasani
approved these changes
Jun 2, 2022
| assertEquals(1, orphanRegionsOnFS.size()); | ||
| assertNull(orphanRegionsOnFS.get(FAKE_HRI.getEncodedName())); | ||
| assertNotNull(orphanRegionsOnFS.get(FAKE_HRI_3.getEncodedName())); | ||
| List holes = (List) result.get(MasterHbckServlet.HOLES); |
Contributor
There was a problem hiding this comment.
Given that we are just comparing null vs non-null and size of the list, this raw use of List is fine.
ndimiduk
reviewed
Jun 6, 2022
ndimiduk
left a comment
Member
There was a problem hiding this comment.
I think this is useful information to expose. I tentatively agree that exposing a report like this deserves its own endpoint, rather than jamming it into JMX.
On the implementation, I'm disappointed to see programming against the raw Servlet API. I did all the work to get Jersey setup and working properly, I hoped that it would be the path forward for new web service work. So I'm -0 on more hand-rolled Servlets.
As for the implementation itself, I think you could a do a little more around error handling. I have a question about using a 503 for an expected server state. In particular, I just got bit by a web UI not handling backup master properly, so that scenario is on my mind during this review.
| private static final Gson GSON = GsonFactory.buildGson(); | ||
| @Override | ||
| public void doGet(HttpServletRequest request, HttpServletResponse response) |
Member
There was a problem hiding this comment.
Why use the raw Servlet API when we have Jersey? Instead of building a response by hand, you could instantiate a POJO and return it, be done.
ContributorAuthor
There was a problem hiding this comment.
The other servlets are implemented using the Servlet API and I used them as template.
There was a problem hiding this comment.
I agree to Nick’s comment on the REST Endpoint returning JSON backed my Model which is a standard way of implementing REST APIs.
| protected void configureInfoServer(InfoServer infoServer) { | ||
| infoServer.addUnprivilegedServlet("master-status", "/master-status", MasterStatusServlet.class); | ||
| infoServer.addUnprivilegedServlet("api_v1", "/api/v1/*", buildApiV1Servlet()); | ||
| infoServer.addUnprivilegedServlet("hbck", "/hbck", MasterHbckServlet.class); |
Member
There was a problem hiding this comment.
Why add a new endpoint instead of having the existing /hbck.jsp honor an "application/json" header and render its response there? Probably a bunch of that jsp's POJOs can be reused...
ContributorAuthor
There was a problem hiding this comment.
I did not and still do not know how to make a JSP page honor different content-encodings. JSP is not the tool I would like to implement this with. If this is the preferred implementation that's fine but I would not want to pursue it.
| final HbckChore hbckChore = master.getHbckChore(); | ||
| if (hbckChore == null || hbckChore.isDisabled()) { | ||
| LOG.warn("Hbck chore is disabled"); | ||
| sendError(response, HttpServletResponse.SC_SERVICE_UNAVAILABLE, "Hbck chore is disabled"); |
Member
There was a problem hiding this comment.
Is this really a 503? The server is fine. I'm no HTTP status code expert, but I'd say this is more like a 204 SC_NO_CONTENT or a 404 SC_NOT_FOUND meaning "hbck resource is not available".
ContributorAuthor
There was a problem hiding this comment.
If the HBCK chore is not available the service is not available, was my thinking. However I can see 204 being quite appropriate.
There was a problem hiding this comment.
404 is usually for resource not found right - if hbck is the resource which is disabled, it means the requested resource is not found.
204 would mean that the resource is found but didn't have content to respond with.
May 404 is right? Given the message contains details that chore is disabled?
| public void doGet(HttpServletRequest request, HttpServletResponse response) | ||
| throws ServletException, IOException { | ||
| final HMaster master = (HMaster) getServletContext().getAttribute(HMaster.MASTER); | ||
| if (!master.isInitialized()) { |
Member
There was a problem hiding this comment.
!isInitialized() could mean that this master instance is a backup master, in which case the response should be a redirect to this endpoint on the active master.
ContributorAuthor
There was a problem hiding this comment.
Ah, this is not something I thought about. Seems reasonable to handle it with a redirect.
| LOG.warn("Failed generating a new catalogjanitor report; using cache", se); | ||
| } | ||
| } | ||
| Map<String, Object> result = new HashMap<>(); |
Member
There was a problem hiding this comment.
If you define a POJO for the response object, we can have some hope of tracing API compatibility in the future using our existing compatibility report.
ContributorAuthor
There was a problem hiding this comment.
Got it, this is a good reason.
There was a problem hiding this comment.
With POJO, we can achieve something like what OData does to even query for a sub document as mentioned here. I am not sure if there is a right equivalent support already available in HBase, else we can skip the OData part but the POJO adds value.
| } | ||
| Map<String, Object> result = new HashMap<>(); | ||
| result.put(START_TIMESTAMP, hbckChore.getCheckingStartTimestamp()); | ||
| result.put(END_TIMESTAMP, hbckChore.getCheckingEndTimestamp()); |
Member
There was a problem hiding this comment.
Our HBCK chore should expose a report so that a reader can get a static view... or a read-write lock so that a reader can block a writer from clobbering its view.
| } | ||
| final CatalogJanitor janitor = master.getCatalogJanitor(); | ||
| if (janitor != null) { | ||
| final Report report = janitor.getLastReport(); |
Member
There was a problem hiding this comment.
CatalogueJanitor exposes the report concept. Brilliant!
ContributorAuthor
There was a problem hiding this comment.
Yes, and unfortunately the result is a mix of CatalogJanitor's nicely structured Report and HbckChore's other return types.
Unifying these things is out of scope, though. Or, we can make it a precondition, but I won't pursue it at this time.
ContributorAuthor
There was a problem hiding this comment.
Down review I accepted your proposal to implement this with Jersey, so a clean unification of these two data sources can be done when working up the model (POJO) of the report encoding to return from the request handler.
| @SuppressWarnings({ "unchecked", "rawtypes" }) | ||
| @Test | ||
| public void testHbckServletWithMocks() throws Exception { |
| doReturn(hbckChore).when(master).getHbckChore(); | ||
| MasterRpcServices services = mock(MasterRpcServices.class); | ||
| doReturn(services).when(master).getRpcServices(); | ||
| doReturn(services).when(master).getMasterRpcServices(); |
Member
There was a problem hiding this comment.
Why do all this mockery? At this point, it's it simpler to stand up a mini-cluster and manufacture the state you need? It looks like TestMetaFixer has some utility code for manufacturing faulty states.
ContributorAuthor
There was a problem hiding this comment.
Because when I wrote this test this was the way I understood it could be implemented and I don't think too much is mocked here, just what is needed to produce a report that has exactly what I want to check for later.
ndimiduk added a commit
to ndimiduk/hbase
that referenced
this pull request
Jun 7, 2022
In apache#4470 for HBASE-26192, it was noted that the HbckChore is kind of a pain to use and test because it maintains a bunch of local state. By contract, the CatalogJanitorChore makes a nice self-contained report. Let's update HbckChore to do the same.
ndimiduk
commented
Jun 7, 2022
Member
I posted up #4498, which will give you a report to work from. |
ContributorAuthor
apurtellforce-pushed
the
HBASE-26192
branch
2 times, most recently
from
June 8, 2022 01:32
cb11a74 to
f8892b0CompareApache-HBase
commented
Jun 8, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 8, 2022
🎊 +1 overall
This message was automatically generated. |
ndimiduk added a commit
to ndimiduk/hbase
that referenced
this pull request
Jun 9, 2022
In apache#4470 for HBASE-26192, it was noted that the HbckChore is kind of a pain to use and test because it maintains a bunch of local state. By contract, the CatalogJanitorChore makes a nice self-contained report. Let's update HbckChore to do the same. Signed-off-by: Andrew Purtell <apurtell@apache.org>
ndimiduk added a commit
to ndimiduk/hbase
that referenced
this pull request
Jun 9, 2022
In apache#4470 for HBASE-26192, it was noted that the HbckChore is kind of a pain to use and test because it maintains a bunch of local state. By contract, the CatalogJanitorChore makes a nice self-contained report. Let's update HbckChore to do the same. Signed-off-by: Andrew Purtell <apurtell@apache.org>
ndimiduk added a commit
to ndimiduk/hbase
that referenced
this pull request
Jun 9, 2022
In apache#4470 for HBASE-26192, it was noted that the HbckChore is kind of a pain to use and test because it maintains a bunch of local state. By contract, the CatalogJanitorChore makes a nice self-contained report. Let's update HbckChore to do the same. Signed-off-by: Andrew Purtell <apurtell@apache.org>
ndimiduk added a commit
to ndimiduk/hbase
that referenced
this pull request
Jun 9, 2022
In apache#4470 for HBASE-26192, it was noted that the HbckChore is kind of a pain to use and test because it maintains a bunch of local state. By contract, the CatalogJanitorChore makes a nice self-contained report. Let's update HbckChore to do the same. Signed-off-by: Andrew Purtell <apurtell@apache.org>
ndimiduk added a commit
that referenced
this pull request
Jun 10, 2022
In #4470 for HBASE-26192, it was noted that the HbckChore is kind of a pain to use and test because it maintains a bunch of local state. By contract, the CatalogJanitorChore makes a nice self-contained report. Let's update HbckChore to do the same. Signed-off-by: Andrew Purtell <apurtell@apache.org>
ndimiduk added a commit
that referenced
this pull request
Jun 10, 2022
In #4470 for HBASE-26192, it was noted that the HbckChore is kind of a pain to use and test because it maintains a bunch of local state. By contract, the CatalogJanitorChore makes a nice self-contained report. Let's update HbckChore to do the same. Signed-off-by: Andrew Purtell <apurtell@apache.org>
ndimiduk added a commit
that referenced
this pull request
Jun 10, 2022
In #4470 for HBASE-26192, it was noted that the HbckChore is kind of a pain to use and test because it maintains a bunch of local state. By contract, the CatalogJanitorChore makes a nice self-contained report. Let's update HbckChore to do the same. Signed-off-by: Andrew Purtell <apurtell@apache.org>
ndimiduk added a commit
that referenced
this pull request
Jun 10, 2022
In #4470 for HBASE-26192, it was noted that the HbckChore is kind of a pain to use and test because it maintains a bunch of local state. By contract, the CatalogJanitorChore makes a nice self-contained report. Let's update HbckChore to do the same. Signed-off-by: Andrew Purtell <apurtell@apache.org>
wenwj0 pushed a commit
to wenwj0/hbase
that referenced
this pull request
Jun 14, 2022
In apache#4470 for HBASE-26192, it was noted that the HbckChore is kind of a pain to use and test because it maintains a bunch of local state. By contract, the CatalogJanitorChore makes a nice self-contained report. Let's update HbckChore to do the same. Signed-off-by: Andrew Purtell <apurtell@apache.org>
Apache-HBase
commented
Jun 17, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 17, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 17, 2022
💔 -1 overall
This message was automatically generated. |
GSON won't deserialize back into HBase native types so don't give the wrong impression by using them.
Apache-HBase
commented
Jun 26, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 26, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 26, 2022
💔 -1 overall
This message was automatically generated. |
vinayakphegde pushed a commit
to vinayakphegde/hbase
that referenced
this pull request
Apr 4, 2024
In apache#4470 for HBASE-26192, it was noted that the HbckChore is kind of a pain to use and test because it maintains a bunch of local state. By contract, the CatalogJanitorChore makes a nice self-contained report. Let's update HbckChore to do the same. Signed-off-by: Andrew Purtell <apurtell@apache.org> Change-Id: I69c1bf867b09b5b9b0d38f69f7cc37c8b48955d1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a trivial recast of hbck.jsp as a servlet. It takes the same actions, using the same parameters, and instead of formatting for HTML constructs a simple map that is formatted as JSON via GSON. Nouns, inclusion, and ordering are the same as that of the jsp page.
It runs as a read-only unprivileged servlet on the master at
/hbck. Hbck JSP pretty printer remains at/hbck.jsp.