Uh oh!
There was an error while loading. Please reload this page.
Update TaskInputOutputContext.java javadoc - #4
Closed
sebastiancadena wants to merge 1 commit into
Closed
Conversation
Update javadoc to reflect the current code structure.
ashishujjain pushed a commit
to odpi/hadoop
that referenced
this pull request
Sep 29, 2015
BUG-39626 Publish source jar for all the HDP stack components.
mekasone pushed a commit
to mekasone/hadoop
that referenced
this pull request
Feb 19, 2017
tangzhankun pushed a commit
to tangzhankun/hadoop
that referenced
this pull request
Feb 24, 2017
Distribute bridge and runtime libraries, and code reformat
isimonenko added a commit
to metamx/hadoop
that referenced
this pull request
Sep 15, 2017
jebat9999
commented
Dec 20, 2017
Better you asked another persons |
jebat9999
approved these changes
Dec 20, 2017
hadoop-yetus
commented
Jul 19, 2019
💔 -1 overall
This message was automatically generated. |
hadoop-yetus
commented
Jul 26, 2019
💔 -1 overall
This message was automatically generated. |
hadoop-yetus
commented
Aug 2, 2019
💔 -1 overall
This message was automatically generated. |
hadoop-yetus
commented
Aug 8, 2019
💔 -1 overall
This message was automatically generated. |
hadoop-yetus
commented
Aug 16, 2019
💔 -1 overall
This message was automatically generated. |
hadoop-yetus
commented
Aug 20, 2019
💔 -1 overall
This message was automatically generated. |
hadoop-yetus
commented
Aug 22, 2019
💔 -1 overall
This message was automatically generated. |
hadoop-yetus
commented
Aug 27, 2019
💔 -1 overall
This message was automatically generated. |
hadoop-yetus
commented
Aug 4, 2021
💔 -1 overall
This message was automatically generated. |
qinghui-xu pushed a commit
to qinghui-xu/hadoop
that referenced
this pull request
Feb 4, 2022
* Fix HDP2 Tokens verification HDP3 is able to read various token that have been migrated to protobuf in the old format, but during sasl verification steps, the receiver (HDPv3.3) verifies the token was correctly encoded by the sender (HPDv2.6). To do so it serializes the token from the extracted fields, using the new format (proto). Due to that, the encoded token by the receiver does not match the one encoded by the sender and the communication fails with a security issue. This commit solves the issue by: - setting a flag to remember if a token was extracted from an old binary format - if the flag is set, when the token is serialized, it is serialized using the old format The commit only covers proto token that can be sent by HDP2 * Remove useless imports Co-authored-by: William Montaz <w.montaz@criteo.com>
Yifan122 added a commit
to Yifan122/hadoop-apache
that referenced
this pull request
Apr 22, 2023
NyteKnight pushed a commit
to NyteKnight/hadoop
that referenced
this pull request
Jun 25, 2024
…ache#4… (apache#356) * HADOOP-18365. Update the remote address when a change is detected (apache#4692) (apache#4768) Back port to branch-3.3, to avoid reconnecting to the old address after detecting that the address has been updated. * Use a stable hashCode to allow safe IP addr changes * Add test that updated address is used Once the address has been updated, it will be used in future calls. Test verifies that a second request succeeds and that it uses the existing updated address instead of having to re-resolve. Co-authored-by: Steve Vaughan Jr <s_vaughan@apple.com> * ACLOVERRIDE --------- Co-authored-by: Steve Vaughan <email@stevevaughan.me> Co-authored-by: Steve Vaughan Jr <s_vaughan@apple.com>
singer-bin pushed a commit
to singer-bin/hadoop
that referenced
this pull request
Dec 19, 2024
…ers on columns This is a re-opened version of: Parquet/parquet-mr#412 The idea behind this pull request is to add a way to express filters on columns using DSL that allows parquet visibility into what is being filtered and how. This visibility will allow us to make optimizations at read time, the biggest one being filtering entire row groups or pages of records without even reading them based on the statistics / metadata that is stored along with each row group or page. Included in this api are interfaces for user defined predicates, which must operate at the value level by may opt in to operating at the row group / page level as well. This should make this new API a superset of the `parquet.filter` package. This new api will need to be reconciled with the column filters currently in the `parquet.filter` package, but I wanted to get feedback on this first. A limitation in both this api and the old one is that you can't do cross-column filters, eg: columX > columnY. Author: Alex Levenson <alexlevenson@twitter.com> Closesapache#4 from isnotinvain/alexlevenson/filter-api and squashes the following commits: c1ab7e3 [Alex Levenson] Address feedback c1bd610 [Alex Levenson] cleanup dotString in ColumnPath 418bfc1 [Alex Levenson] Update version, add temporary hacks for semantic enforcer 6643bd3 [Alex Levenson] Fix some more non backward incompatible changes 39f977f [Alex Levenson] Put a bunch of backwards compatible stuff back in, add @deprecated 13a02c6 [Alex Levenson] Fix compile errors, add back in overloaded getRecordReader f82edb7 [Alex Levenson] Merge branch 'master' into alexlevenson/filter-api 9bd014f [Alex Levenson] clean up TODOs and reference jiras 4cc7e87 [Alex Levenson] Add some comments 30e3d61 [Alex Levenson] Create a common interface for both kinds of filters ac153a6 [Alex Levenson] Create a Statistics class for use in UDPs fbbf601 [Alex Levenson] refactor IncrementallyUpdatedFilterPredicateGenerator to only generate the parts that require generation 5df47cd [Alex Levenson] Static imports of checkNotNull c1d1823 [Alex Levenson] address some of the minor feedback items 67a3ba0 [Alex Levenson] update binary's toString 3d7372b [Alex Levenson] minor fixes fed9531 [Alex Levenson] Add skipCurrentRecord method to clear events in thrift converter 2e632d5 [Alex Levenson] Make Binary Serializable 09c024f [Alex Levenson] update comments 3169849 [Alex Levenson] fix compilation error 0185030 [Alex Levenson] Add integration test for value level filters 4fde18c [Alex Levenson] move to right package ae36b37 [Alex Levenson] Handle merge issues af69486 [Alex Levenson] Merge branch 'master' into alexlevenson/filter-api 0665271 [Alex Levenson] Add tests for value inspector c5e3b07 [Alex Levenson] Add tests for resetter and evaluator 29f677a [Alex Levenson] Fix scala DSL 8897a28 [Alex Levenson] Fix some tests b448bee [Alex Levenson] Fix mistake in MessageColumnIO c8133f8 [Alex Levenson] Fix some tests 4cf686d [Alex Levenson] more null checks 69e683b [Alex Levenson] check all the nulls 220a682 [Alex Levenson] more cleanup aad5af3 [Alex Levenson] rm generated src file from git 5075243 [Alex Levenson] more minor cleanup 9966713 [Alex Levenson] Hook generation into maven build 8282725 [Alex Levenson] minor cleanup fea3ea9 [Alex Levenson] minor cleanup 9e35406 [Alex Levenson] move statistics filter c52750c [Alex Levenson] finish moving things around 97a6bfd [Alex Levenson] Move things around pt2 843b9fe [Alex Levenson] Move some files around pt 1 5eedcc0 [Alex Levenson] turn off dictionary support for AtomicConverter 541319e [Alex Levenson] various cleanup and fixes 08e9638 [Alex Levenson] rm ColumnPathUtil bfe6795 [Alex Levenson] Add type bounds to FilterApi 6c831ab [Alex Levenson] don't double log exception in SerializationUtil a7a58d1 [Alex Levenson] use ColumnPath instead of String 8f11a6b [Alex Levenson] Move ColumnPath and Canonicalizer to parquet-common 9164359 [Alex Levenson] stash abc2be2 [Alex Levenson] Add null handling to record filters -- this impl is still broken though 90ba8f7 [Alex Levenson] Update Serialization Util 0a261f1 [Alex Levenson] Add compression in SerializationUtil f1278be [Alex Levenson] Add comment, fix tests cbd1a85 [Alex Levenson] Replace some specialization with generic views e496cbf [Alex Levenson] Fix short circuiting in StatisticsFilter db6b32d [Alex Levenson] Address some comments, fix constructor in ParquetReader fd6f44d [Alex Levenson] Fix semver backward compat 2fdd304 [Alex Levenson] Some more cleanup d34fb89 [Alex Levenson] Cleanup some TODOs 544499c [Alex Levenson] stash 7b32016 [Alex Levenson] Merge branch 'master' into alexlevenson/filter-api 0e31251 [Alex Levenson] First pass at values filter, needs reworking 470e409 [Alex Levenson] fix java6/7 bug, minor cleanup ee7b221 [Alex Levenson] more InputFormat tests 5ef849e [Alex Levenson] Add guards for not specifying both kinds of filter 0186b1f [Alex Levenson] Add logging to ParquetInputFormat and tests for configuration a622648 [Alex Levenson] cleanup imports 9b1ea88 [Alex Levenson] Add tests for statistics filter d517373 [Alex Levenson] tests for filter validator b25fc44 [Alex Levenson] small cleanup of filter validator 32067a1 [Alex Levenson] add test for collapse logical nots 1efc198 [Alex Levenson] Add tests for invert filter predicate 046b106 [Alex Levenson] some more fixes d3c4d7a [Alex Levenson] fix some more types, add in test for SerializationUtil cc51274 [Alex Levenson] fix generics in FilterPredicateInverter ea08349 [Alex Levenson] First pass at rowgroup filter, needs testing 156d91b [Alex Levenson] Add runtime type checker 4dfb4f2 [Alex Levenson] Add serialization util 8f80b20 [Alex Levenson] update comment 7c25121 [Alex Levenson] Add class to Column struct 58f1190 [Alex Levenson] Remove filterByUniqueValues 7f20de6 [Alex Levenson] rename user predicates af14b42 [Alex Levenson] Update dsl 04409c5 [Alex Levenson] Add generic types into Visitor ba42884 [Alex Levenson] rm getClassName 65f8af9 [Alex Levenson] Add in support for user defined predicates on columns 6926337 [Alex Levenson] Add explicit tokens for notEq, ltEq, gtEq 667ec9f [Alex Levenson] remove test for collapsing double negation db2f71a [Alex Levenson] rename FilterPredicatesTest a0a0533 [Alex Levenson] Address first round of comments b2bca94 [Alex Levenson] Add scala DSL and tests bedda87 [Alex Levenson] Add tests for FilterPredicate building 238cbbe [Alex Levenson] Add scala dsl 39f7b24 [Alex Levenson] add scala mvn boilerplate 2ec71a7 [Alex Levenson] Add predicate API
singer-bin pushed a commit
to singer-bin/hadoop
that referenced
this pull request
Dec 19, 2024
Author: asingh <asingh@cloudera.com> Author: Alex Levenson <alexlevenson@twitter.com> Author: Ashish Singh <asingh@cloudera.com> Closesapache#197 from SinghAsDev/PARQUET-251 and squashes the following commits: 68e0eae [asingh] Remove deprecated constructors from private classes 67e4e5f [asingh] Add removed public methods in Binary and deprecate them 0e71728 [asingh] Add comment for BinaryStatistics.setMinMaxFromBytes fbe873f [Ashish Singh] Merge pull request apache#4 from isnotinvain/PR-197-3 9826ee6 [Alex Levenson] Some minor cleanup 7570035 [asingh] Remove test for stats getting ingnored for version 160 when type is int64 af43d28 [Alex Levenson] Address PR feedback 89ab4ee [Alex Levenson] put the headers in the right location 2838cc9 [Alex Levenson] Split out version checks to separate files, add some tests 5af9142 [Alex Levenson] Generalize tests, make Binary.fromString reused=false e00d9b7 [asingh] Rename isReused => isBackingBytesReused d2ad939 [asingh] Rebase over latest trunk 857141a [asingh] Remove redundant junit dependency 32b88ed [asingh] Remove semver from hadoop-common 7a0e99e [asingh] Revert to fromConstantByteArray for ByteString c820ec9 [asingh] Add unit tests for Binary and to check if stats are ignored for version 160 9bbd1e5 [asingh] Improve version parsing 84a1d8b [asingh] Remove ignoring stats on write side and ignore it on read side 903f8e3 [asingh] Address some review comments. * Ignore stats for writer's version < 1.8.0 * Refactor shoudlIgnoreStatistics method a bit * Assume implementations other than parquet-mr were writing binary statistics correctly * Add toParquetStatistics method's original method signature to maintain backwards compatibility and mark it as deprecated 64c2617 [asingh] Revert changes for ignoring stats at RowGroupFilter level e861b18 [asingh] Ignore max min stats while reading 3a8cb8d [asingh] Fix typo 8e12618 [asingh] Fix usage of fromConstant versions of Binary constructors 860adf7 [asingh] Rename unmodified to constant and isReused instead of isUnmodifiable 0d127a7 [asingh] Add unmodfied and Reused versions for creating a Binary. Add copy() to Binary. b4e2950 [asingh] Skip filtering based on stats when file was written with version older than 1.6.1 6fcee8c [asingh] Add getBytesUnsafe() to Binary that returns backing byte[] if possible, else returns result of getBytes() 30b07dd [asingh] PARQUET-251: Binary column statistics error when reuse byte[] among rows
Contributor
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
LuciferYang added a commit
to LuciferYang/hadoop
that referenced
this pull request
Apr 8, 2026
…irectoryException fix.
Round 8 pivoted from visual code review to empirical testing — running
existing HDFS test classes against IIPBasedFSNamesystemLock to find
dormant bugs that visual review can miss. Added two artifacts:
1. TestFSNamesystemFGLIIPStress (7 stress tests):
- concurrentRenameAndGetFileInfo: rename + getFileInfo soak (2s)
- concurrentCreateAndDelete: create+delete race tolerance (race-
related exceptions like FileNotFoundException on complete are
accepted as legitimate HDFS semantics)
- saveNamespaceUnderLoad: saveNamespace + restartNameNode while
a worker drives create+getFileInfo, verify post-restart works
- concurrentSetPermissionAndGetFileInfo: legacy setPermission
(compat-write) vs pilot getFileInfo (compat-read) mutual
exclusion soak
- mixedPilotAndLegacySoak: 4 pilot creators + 4 legacy operators
mixing mkdir/delete/setPermission for 3s
- quotaBearingDirectoryFallsBackToLegacy: setQuota then create
verifies envelope-miss + legacy fallback path
- massFileCreation: 10 dirs x 50 files = 500 concurrent creates
2. Test-only system property override in FSNamesystem:
System.getProperty('test.dfs.namenode.lock.model.provider.class')
overrides the configured lock model. Lets us run any existing
HDFS test class against IIPBasedFSNamesystemLock with:
-Dtest.dfs.namenode.lock.model.provider.class=...IIPBasedFSNamesystemLock
BUG apache#4 (round 8, MEDIUM): U8a getFileInfoPilot did not convert
ParentNotDirectoryException to AccessControlException
Symptom: TestDFSPermission#testPermissionMessageOnNonDirAncestor
failed under FGL_IIP. The test calls exists('/p4/file/nonexisting')
on a path where /p4/file is a regular file (not a directory).
Legacy path:
1. FSDirectory.resolvePath -> checkTraverse throws
ParentNotDirectoryException for the non-dir ancestor
2. resolvePath catches PNDE and converts to ACE for non-create
operations (FSDirectory.java:752-758)
3. FSDirStatAndListingOp.getFileInfo catches ACE for superuser
and returns null
4. exists() converts null to false
Pilot path (before fix):
1. dir.checkTraverse(pc, lip.iip(), dirOp) throws PNDE
2. My catch handled AccessControlException only — PNDE
propagated as IOException with message '/p4/file (is not a
directory)'
3. Test failed with the wrong exception type
Fix: add a sibling catch for ParentNotDirectoryException in
getFileInfoPilot. For superuser: return null (matches legacy).
For non-superuser: convert to AccessControlException with the
PNDE message preserved (matches legacy resolvePath conversion).
Test verification under FGL_IIP after the fix:
TestDFSPermission 10 tests, all pass
TestFileCreation 25 tests, all pass (2 env-skipped)
TestQuota 23 tests, all pass
TestFsck 36 tests, all pass
TestDFSRename 3 tests, all pass
TestSnapshotFileLength 2 tests, all pass
Plus all 145 pilot tests (TestFSNamesystemFGLIIPStress + others)
pass.
Discovered during round-8 empirical review. Three rounds of
empirical / implementation review (rounds 6, 7, 8) have now found a
total of 5 real bugs that visual design review (rounds 1-5) did not.
This validates the round-7 prediction: empirical testing on real
workloads is higher-yield than another visual code review.
JIRA: HDFS-17385
Affected: U8a (getFileInfoPilot) and FSNamesystem test overrideLuciferYang added a commit
to LuciferYang/hadoop
that referenced
this pull request
Apr 16, 2026
Third post-U8 RPC migration. Uses PATH_READ — same mode as getFileInfo and getBlockLocations. This is the simplest read-path migration so far: 2-line work inside the held lock, no BM interaction, no access-time side effect. Envelope: shared canUsePilotPathRead check (see below). Phase B (under PATH_READ): - hand-over-hand walk populates the IIP with per-INode read locks - checkTraverse on the IIP (mirrors legacy fsd.resolvePath's permission side; PNDE → ACE conversion for non-superuser matches getFileInfo pilot's Round-8 BUG apache#4 fix) - INodeFile.valueOf validates target (FileNotFoundException when absent, IOException when directory) — legacy parity - return !isUnderConstruction() - InterruptedException wrapped to InterruptedIOException Shared Phase-A envelope helper: Extracted canUsePilotPathRead(String src) in FSNamesystem. Both getFileInfo and getBlockLocations pilot paths had identical Phase-A bodies (fsLock type, src validity, /.reserved, /.snapshot, provider != null). isFileClosed would have been the third copy; extracting now prevents the third-copy regret and does not wait for the §6.3 RPC-10 gate because the duplication was already concrete, not speculative. Both callers updated. Net effect on behaviour for getFileInfo: it now also rejects when a KeyProvider is configured — previously unguarded. No test sets up a provider, and the stricter guard matches getBlockLocations/isFileClosed; if a future encrypted-cluster workload requires narrower gating, we can split the helpers. Code shape: - FSDirStatAndListingOp.isFileClosed(fsd, iip, src) — new pre-resolved-IIP overload, parallel to the existing FSDirStatAndListingOp.getFileInfo(fsd, iip, ...) overload. - FSNamesystem.isFileClosedPilot(src, pc) — PATH_READ + checkTraverse + pre-resolved-IIP call. - FSNamesystem.canUsePilotPathRead(src) — shared Phase-A check. Tests (7 new in TestFSNamesystemFGLIIP): - isFileClosedOnClosedFile — happy path - isFileClosedWhileUnderConstruction — writer still open, then closed; pilot must observe both states - isFileClosedOnMissingFileFails — FileNotFoundException - isFileClosedOnDirectoryFails — IOException via INodeFile.valueOf - isFileClosedSnapshotPathFallsBackToLegacy - isFileClosedRespectsTraversalPermission — non-owner w/o x on ancestor gets ACE - parallelIsFileClosedOnSameFile — 16 threads × 100 on hot file Verification: mvn test -pl hadoop-hdfs-project/hadoop-hdfs -o \ -Dtest='TestFSNamesystemFGLIIP' => Tests run: 40, Failures: 0, Errors: 0, Skipped: 0 mvn test -pl hadoop-hdfs-project/hadoop-hdfs -o -Dtest=\ 'TestFSNamesystemFGLIIP,TestIIPBasedFSNamesystemLock,TestLockPool,\ TestLockRef,TestIIPAcquireMode,TestLockedIIP,TestINodeLockManager,\ TestINodeMapConcurrent,TestGetFullPathNameConcurrent' => Tests run: 165, Failures: 0, Errors: 0, Skipped: 0 RPC-10 gate counter: 3 / 10. Spec: docs/fgl/HDFS-17385-wave4-pilot-design.md §2.9, §3.1 Checklist: docs/fgl/rpc-migration-checklist.md
LuciferYang added a commit
to LuciferYang/hadoop
that referenced
this pull request
Apr 16, 2026
Ninth RPC migrated. Fourth PATH_READ RPC — first to exercise directory iteration and pagination under held per-INode read locks. RPC-10 gate counter: 9/10. Pausing here for the pre-gate review before the 10th RPC triggers the §6.3 refactor gate. Envelope (Phase A, lock-free): - canUsePilotPathRead (shared helper) — fsLock, reserved, snapshot, encryption-provider checks - startAfter is not INodePath-style (starts with '/') — legacy resolveComponents-based resolution is out of pilot scope canUsePilotGetListing composes canUsePilotPathRead with the extra startAfter check. Phase B (under PATH_READ): - checkTraverse on the IIP (mirrors legacy resolvePath permission side; PNDE → ACE for non-superuser, matches getFileInfo / isFileClosed pilot Round-8 BUG apache#4 fix) - delegates to FSDirStatAndListingOp.getListingForPilot pre-resolved-IIP overload which runs the directory-target READ_EXECUTE permission check and calls the existing private getListing - null return from pilot is a legitimate result (target not found, single-file listing, etc.) — unlike other pilot RPCs where null means "envelope miss, fall back". Envelope misses are signalled exclusively via PilotEnvelopeMissException. Code shape: - FSDirStatAndListingOp.getListingForPilot(fsd, pc, iip, startAfter, needLocation) — new pre-resolved-IIP overload wrapping the existing private getListing(fsd, iip, …). - FSNamesystem.canUsePilotGetListing — composes the shared canUsePilotPathRead with the startAfter envelope check. - FSNamesystem.getListingPilot — standard PATH_READ + checkTraverse + delegation. Consistency semantics documented: Directory iteration reads child INode metadata without taking per-child locks. A concurrent PATH_WRITE on a child could race with the listing's read of that child's metadata. The resulting "slightly stale" data is equivalent to legacy's "locked-then- released-then-client-reads-the-result" behaviour. Fields read (permission, mtime, atime) are primitive long or packed long — no torn-read risk at the field level. Clients already expect listings to be point-in-time snapshots that become stale immediately; pilot preserves this contract. BM lock handling: The existing private createFileStatus(fsd, iip, child, storagePolicy, needLocation=true, …) internally acquires BM readLock around the createLocatedBlocks call (line 516-530 of FSDirStatAndListingOp). The pilot does not need to wrap createFileStatus in a separate BM lock — the inner acquisition is per-child, matching the checklist rule 4 (IIPLock > BMLock) because the IIP-side read lock is held throughout the iteration. Tests (10 new in TestFSNamesystemFGLIIP): - getListingOnDirectory — happy path, 5 children - getListingOnFile — single-file listing returns self as a 1-entry listing - getListingOnMissingPathFails — FileNotFoundException - getListingOnEmptyDirectory — 0-length listing - getListingPaginationAcrossBatches — 50 entries iterated via RemoteIterator; exercises startAfter rollover across batches - getListingWithBlockLocations — listLocatedStatus with needLocation=true; verifies each file gets valid block locations - getListingSnapshotPathFallsBackToLegacy — /.snapshot/s1 listing routes through legacy - getListingReservedPathFallsBackToLegacy — /.reserved listing returns the well-known entries - getListingRespectsReadExecutePermission — non-owner without READ_EXECUTE on directory gets ACE - parallelGetListingDisjointDirectories — 16 × 30 on distinct directories Verification: mvn test -pl hadoop-hdfs-project/hadoop-hdfs -o \ -Dtest='TestFSNamesystemFGLIIP' => Tests run: 90, Failures: 0, Errors: 0, Skipped: 0 RPC-10 gate counter: 9/10. The next RPC triggers the §6.3 gate. Pausing here for the pre-gate review commit. Spec: docs/fgl/HDFS-17385-wave4-pilot-design.md §2.9, §3.1 Checklist: docs/fgl/rpc-migration-checklist.md
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update javadoc to reflect the current code structure.