EBSD map without loading a file name

[jorgenasorhaug]

I have issues with reading ctf and ang files saved using Orix and MTEX. I therefore think it would be great to be able to "go around" this by making an EBSD map from the raw data by e.g. reading a numpy file first. However, I'm struggling to figure out how to do so, if possible at all?

[mikesmic]

Your work around would take some effort to implement and I would much prefer to figure out why your files are not loading. Could you send details of the error you get and a small example file that will not load?

[jorgenasorhaug]

I see. Is there then a specific way to write a dictionary file that can read using defdap?

My problems starts here:

File ~\AppData\Local\anaconda3\envs\defdap\Lib\site-packages\defdap\file_readers.py:286, in EdaxAngLoader.load(self, file_name)
283 elif line.startswith('NROWS'):
284 ydim = int(line.split()[-1])
--> 286 shape = (ydim, xdim)
287 self.loaded_metadata['shape'] = shape
289 self.check_metadata()
UnboundLocalError: cannot access local variable 'ydim' where it is not associated with a value

If I manually define these variables in file_readers.py, a different error occurs

File ~\AppData\Local\anaconda3\envs\defdap\Lib\site-packages\numpy\lib\_npyio_impl.py:1046, in _read(fname, delimiter, comment, quote, imaginary_unit, usecols, skiplines, max_rows, converters, ndmin, unpack, dtype, encoding)
1043 data = _preprocess_comments(data, comments, encoding)
1045 if read_dtype_via_object_chunks is None:
-> 1046 arr = _load_from_filelike(
1047 data, delimiter=delimiter, comment=comment, quote=quote,
1048 imaginary_unit=imaginary_unit,
1049 usecols=usecols, skiplines=skiplines, max_rows=max_rows,
1050 converters=converters, dtype=dtype,
1051 encoding=encoding, filelike=filelike,
1052 byte_converters=byte_converters)
1054 else:
1055 # This branch reads the file into chunks of object arrays and then
1056 # casts them to the desired actual dtype. This ensures correct
1057 # string-length and datetime-unit discovery (like `arr.astype()`).
1058 # Due to chunking, certain error reports are less clear, currently.
1059 if filelike:
TypeError: If a structured dtype is used, the number of columns in `usecols` must match the effective number of fields. But 8 usecols were given and the number of fields is 7.

You can test this dummy dataset
024 xmap_hi.zip

[mikesmic]

.ang files seem to be a mess and not very well defined. I've made a small change on the develop branch that now seems to load your file. You will have to make a small modification to the file though. Please add an empty commented line between NumberFamilies and GRID:, should look like this after:

# NumberFamilies 0
#
# GRID: SqrGrid

Only the first 8 columns of data will be loaded in, as beyond those it seems to a complete guess as to what the data is in all the different .ang formats. Hope this helps, let me know how you get on.

[jorgenasorhaug]

This worked great, thanks! However, I had to change the 'phase' datatype in file_readers.py from uint8 to int8 to read -1 (not indexed): self.data_format = np.dtype([ ... ('phase', 'int8'),])

Further, I see that the reference frames are different between kikuchipy (which I've used to index the patterns) and DefDAP. Is there a way to change the reference frame or rotate my data using DefDAP?

[mikesmic]

Is -1 for non-indexed standard for .ang file? I think the loader currently uses the same convention as the oxford formats, 0 for non-indexed and phases start at 1.

Defdap uses the same convention as oxford ebsd systems for reference frames (spatial frame is x right, y down, z into page and euler angle frame x right, y up, z out). EDAX uses the same convention for spatial frame but the euler frame is rotated 90 deg around z for x down, y right, z out. A correction is applied to the Euler angles when loading .ang files (

DefDAP/defdap/file_readers.py

Line 342 in 4e6d9d6

euler_angle[0] -=np.pi/2

), so you could try changing this to make everything consistent for your input file and we can maybe make the correction a input parameter for the loader.

[jorgenasorhaug]

I'm not sure what the standard of .ang files are, but the python package kikuchipy uses the package orix to make the orientation maps. Orix labels (and saves) the not-indexed pixels as -1.

Thanks for the suggestion. I commented out the line and it seems to be giving me the expected orientation :)

[jorgenasorhaug]

I have issues with reading ctf and ang files saved using Orix and MTEX. I therefore think it would be great to be able to "go around" this by making an EBSD map from the raw data by e.g. reading a numpy file first. However, I'm struggling to figure out how to do so, if possible at all?

[mikesmic]

Your work around would take some effort to implement and I would much prefer to figure out why your files are not loading. Could you send details of the error you get and a small example file that will not load?

[jorgenasorhaug]

I see. Is there then a specific way to write a dictionary file that can read using defdap?

My problems starts here:

File ~\AppData\Local\anaconda3\envs\defdap\Lib\site-packages\defdap\file_readers.py:286, in EdaxAngLoader.load(self, file_name)
283 elif line.startswith('NROWS'):
284 ydim = int(line.split()[-1])
--> 286 shape = (ydim, xdim)
287 self.loaded_metadata['shape'] = shape
289 self.check_metadata()
UnboundLocalError: cannot access local variable 'ydim' where it is not associated with a value

If I manually define these variables in file_readers.py, a different error occurs

File ~\AppData\Local\anaconda3\envs\defdap\Lib\site-packages\numpy\lib\_npyio_impl.py:1046, in _read(fname, delimiter, comment, quote, imaginary_unit, usecols, skiplines, max_rows, converters, ndmin, unpack, dtype, encoding)
1043 data = _preprocess_comments(data, comments, encoding)
1045 if read_dtype_via_object_chunks is None:
-> 1046 arr = _load_from_filelike(
1047 data, delimiter=delimiter, comment=comment, quote=quote,
1048 imaginary_unit=imaginary_unit,
1049 usecols=usecols, skiplines=skiplines, max_rows=max_rows,
1050 converters=converters, dtype=dtype,
1051 encoding=encoding, filelike=filelike,
1052 byte_converters=byte_converters)
1054 else:
1055 # This branch reads the file into chunks of object arrays and then
1056 # casts them to the desired actual dtype. This ensures correct
1057 # string-length and datetime-unit discovery (like `arr.astype()`).
1058 # Due to chunking, certain error reports are less clear, currently.
1059 if filelike:
TypeError: If a structured dtype is used, the number of columns in `usecols` must match the effective number of fields. But 8 usecols were given and the number of fields is 7.

You can test this dummy dataset
024 xmap_hi.zip

[mikesmic]

.ang files seem to be a mess and not very well defined. I've made a small change on the develop branch that now seems to load your file. You will have to make a small modification to the file though. Please add an empty commented line between NumberFamilies and GRID:, should look like this after:

# NumberFamilies 0
#
# GRID: SqrGrid

Only the first 8 columns of data will be loaded in, as beyond those it seems to a complete guess as to what the data is in all the different .ang formats. Hope this helps, let me know how you get on.

[jorgenasorhaug]

This worked great, thanks! However, I had to change the 'phase' datatype in file_readers.py from uint8 to int8 to read -1 (not indexed): self.data_format = np.dtype([ ... ('phase', 'int8'),])

Further, I see that the reference frames are different between kikuchipy (which I've used to index the patterns) and DefDAP. Is there a way to change the reference frame or rotate my data using DefDAP?

[mikesmic]

Is -1 for non-indexed standard for .ang file? I think the loader currently uses the same convention as the oxford formats, 0 for non-indexed and phases start at 1.

Defdap uses the same convention as oxford ebsd systems for reference frames (spatial frame is x right, y down, z into page and euler angle frame x right, y up, z out). EDAX uses the same convention for spatial frame but the euler frame is rotated 90 deg around z for x down, y right, z out. A correction is applied to the Euler angles when loading .ang files (

DefDAP/defdap/file_readers.py

Line 342 in 4e6d9d6

euler_angle[0] -=np.pi/2

), so you could try changing this to make everything consistent for your input file and we can maybe make the correction a input parameter for the loader.

[jorgenasorhaug]

I'm not sure what the standard of .ang files are, but the python package kikuchipy uses the package orix to make the orientation maps. Orix labels (and saves) the not-indexed pixels as -1.

Thanks for the suggestion. I commented out the line and it seems to be giving me the expected orientation :)

[jorgenasorhaug]

I have issues with reading ctf and ang files saved using Orix and MTEX. I therefore think it would be great to be able to "go around" this by making an EBSD map from the raw data by e.g. reading a numpy file first. However, I'm struggling to figure out how to do so, if possible at all?

[mikesmic]

Your work around would take some effort to implement and I would much prefer to figure out why your files are not loading. Could you send details of the error you get and a small example file that will not load?

[jorgenasorhaug]

I see. Is there then a specific way to write a dictionary file that can read using defdap?

My problems starts here:

File ~\AppData\Local\anaconda3\envs\defdap\Lib\site-packages\defdap\file_readers.py:286, in EdaxAngLoader.load(self, file_name)
283 elif line.startswith('NROWS'):
284 ydim = int(line.split()[-1])
--> 286 shape = (ydim, xdim)
287 self.loaded_metadata['shape'] = shape
289 self.check_metadata()
UnboundLocalError: cannot access local variable 'ydim' where it is not associated with a value

If I manually define these variables in file_readers.py, a different error occurs

File ~\AppData\Local\anaconda3\envs\defdap\Lib\site-packages\numpy\lib\_npyio_impl.py:1046, in _read(fname, delimiter, comment, quote, imaginary_unit, usecols, skiplines, max_rows, converters, ndmin, unpack, dtype, encoding)
1043 data = _preprocess_comments(data, comments, encoding)
1045 if read_dtype_via_object_chunks is None:
-> 1046 arr = _load_from_filelike(
1047 data, delimiter=delimiter, comment=comment, quote=quote,
1048 imaginary_unit=imaginary_unit,
1049 usecols=usecols, skiplines=skiplines, max_rows=max_rows,
1050 converters=converters, dtype=dtype,
1051 encoding=encoding, filelike=filelike,
1052 byte_converters=byte_converters)
1054 else:
1055 # This branch reads the file into chunks of object arrays and then
1056 # casts them to the desired actual dtype. This ensures correct
1057 # string-length and datetime-unit discovery (like `arr.astype()`).
1058 # Due to chunking, certain error reports are less clear, currently.
1059 if filelike:
TypeError: If a structured dtype is used, the number of columns in `usecols` must match the effective number of fields. But 8 usecols were given and the number of fields is 7.

You can test this dummy dataset
024 xmap_hi.zip

[mikesmic]

.ang files seem to be a mess and not very well defined. I've made a small change on the develop branch that now seems to load your file. You will have to make a small modification to the file though. Please add an empty commented line between NumberFamilies and GRID:, should look like this after:

# NumberFamilies 0
#
# GRID: SqrGrid

Only the first 8 columns of data will be loaded in, as beyond those it seems to a complete guess as to what the data is in all the different .ang formats. Hope this helps, let me know how you get on.

[jorgenasorhaug]

This worked great, thanks! However, I had to change the 'phase' datatype in file_readers.py from uint8 to int8 to read -1 (not indexed): self.data_format = np.dtype([ ... ('phase', 'int8'),])

Further, I see that the reference frames are different between kikuchipy (which I've used to index the patterns) and DefDAP. Is there a way to change the reference frame or rotate my data using DefDAP?

[mikesmic]

Is -1 for non-indexed standard for .ang file? I think the loader currently uses the same convention as the oxford formats, 0 for non-indexed and phases start at 1.

Defdap uses the same convention as oxford ebsd systems for reference frames (spatial frame is x right, y down, z into page and euler angle frame x right, y up, z out). EDAX uses the same convention for spatial frame but the euler frame is rotated 90 deg around z for x down, y right, z out. A correction is applied to the Euler angles when loading .ang files (

DefDAP/defdap/file_readers.py

Line 342 in 4e6d9d6

euler_angle[0] -=np.pi/2

), so you could try changing this to make everything consistent for your input file and we can maybe make the correction a input parameter for the loader.

[jorgenasorhaug]

I'm not sure what the standard of .ang files are, but the python package kikuchipy uses the package orix to make the orientation maps. Orix labels (and saves) the not-indexed pixels as -1.

Thanks for the suggestion. I commented out the line and it seems to be giving me the expected orientation :)

[jorgenasorhaug]

I have issues with reading ctf and ang files saved using Orix and MTEX. I therefore think it would be great to be able to "go around" this by making an EBSD map from the raw data by e.g. reading a numpy file first. However, I'm struggling to figure out how to do so, if possible at all?

[mikesmic]

Your work around would take some effort to implement and I would much prefer to figure out why your files are not loading. Could you send details of the error you get and a small example file that will not load?

[jorgenasorhaug]

I see. Is there then a specific way to write a dictionary file that can read using defdap?

My problems starts here:

File ~\AppData\Local\anaconda3\envs\defdap\Lib\site-packages\defdap\file_readers.py:286, in EdaxAngLoader.load(self, file_name)
283 elif line.startswith('NROWS'):
284 ydim = int(line.split()[-1])
--> 286 shape = (ydim, xdim)
287 self.loaded_metadata['shape'] = shape
289 self.check_metadata()
UnboundLocalError: cannot access local variable 'ydim' where it is not associated with a value

If I manually define these variables in file_readers.py, a different error occurs

File ~\AppData\Local\anaconda3\envs\defdap\Lib\site-packages\numpy\lib\_npyio_impl.py:1046, in _read(fname, delimiter, comment, quote, imaginary_unit, usecols, skiplines, max_rows, converters, ndmin, unpack, dtype, encoding)
1043 data = _preprocess_comments(data, comments, encoding)
1045 if read_dtype_via_object_chunks is None:
-> 1046 arr = _load_from_filelike(
1047 data, delimiter=delimiter, comment=comment, quote=quote,
1048 imaginary_unit=imaginary_unit,
1049 usecols=usecols, skiplines=skiplines, max_rows=max_rows,
1050 converters=converters, dtype=dtype,
1051 encoding=encoding, filelike=filelike,
1052 byte_converters=byte_converters)
1054 else:
1055 # This branch reads the file into chunks of object arrays and then
1056 # casts them to the desired actual dtype. This ensures correct
1057 # string-length and datetime-unit discovery (like `arr.astype()`).
1058 # Due to chunking, certain error reports are less clear, currently.
1059 if filelike:
TypeError: If a structured dtype is used, the number of columns in `usecols` must match the effective number of fields. But 8 usecols were given and the number of fields is 7.

You can test this dummy dataset
024 xmap_hi.zip

[mikesmic]

.ang files seem to be a mess and not very well defined. I've made a small change on the develop branch that now seems to load your file. You will have to make a small modification to the file though. Please add an empty commented line between NumberFamilies and GRID:, should look like this after:

# NumberFamilies 0
#
# GRID: SqrGrid

Only the first 8 columns of data will be loaded in, as beyond those it seems to a complete guess as to what the data is in all the different .ang formats. Hope this helps, let me know how you get on.

[jorgenasorhaug]

This worked great, thanks! However, I had to change the 'phase' datatype in file_readers.py from uint8 to int8 to read -1 (not indexed): self.data_format = np.dtype([ ... ('phase', 'int8'),])

Further, I see that the reference frames are different between kikuchipy (which I've used to index the patterns) and DefDAP. Is there a way to change the reference frame or rotate my data using DefDAP?

[mikesmic]

Is -1 for non-indexed standard for .ang file? I think the loader currently uses the same convention as the oxford formats, 0 for non-indexed and phases start at 1.

Defdap uses the same convention as oxford ebsd systems for reference frames (spatial frame is x right, y down, z into page and euler angle frame x right, y up, z out). EDAX uses the same convention for spatial frame but the euler frame is rotated 90 deg around z for x down, y right, z out. A correction is applied to the Euler angles when loading .ang files (

DefDAP/defdap/file_readers.py

Line 342 in 4e6d9d6

euler_angle[0] -=np.pi/2

), so you could try changing this to make everything consistent for your input file and we can maybe make the correction a input parameter for the loader.

[jorgenasorhaug]

I'm not sure what the standard of .ang files are, but the python package kikuchipy uses the package orix to make the orientation maps. Orix labels (and saves) the not-indexed pixels as -1.

Thanks for the suggestion. I commented out the line and it seems to be giving me the expected orientation :)

[jorgenasorhaug]

I have issues with reading ctf and ang files saved using Orix and MTEX. I therefore think it would be great to be able to "go around" this by making an EBSD map from the raw data by e.g. reading a numpy file first. However, I'm struggling to figure out how to do so, if possible at all?

[mikesmic]

Your work around would take some effort to implement and I would much prefer to figure out why your files are not loading. Could you send details of the error you get and a small example file that will not load?

[jorgenasorhaug]

I see. Is there then a specific way to write a dictionary file that can read using defdap?

My problems starts here:

File ~\AppData\Local\anaconda3\envs\defdap\Lib\site-packages\defdap\file_readers.py:286, in EdaxAngLoader.load(self, file_name)
283 elif line.startswith('NROWS'):
284 ydim = int(line.split()[-1])
--> 286 shape = (ydim, xdim)
287 self.loaded_metadata['shape'] = shape
289 self.check_metadata()
UnboundLocalError: cannot access local variable 'ydim' where it is not associated with a value

If I manually define these variables in file_readers.py, a different error occurs

File ~\AppData\Local\anaconda3\envs\defdap\Lib\site-packages\numpy\lib\_npyio_impl.py:1046, in _read(fname, delimiter, comment, quote, imaginary_unit, usecols, skiplines, max_rows, converters, ndmin, unpack, dtype, encoding)
1043 data = _preprocess_comments(data, comments, encoding)
1045 if read_dtype_via_object_chunks is None:
-> 1046 arr = _load_from_filelike(
1047 data, delimiter=delimiter, comment=comment, quote=quote,
1048 imaginary_unit=imaginary_unit,
1049 usecols=usecols, skiplines=skiplines, max_rows=max_rows,
1050 converters=converters, dtype=dtype,
1051 encoding=encoding, filelike=filelike,
1052 byte_converters=byte_converters)
1054 else:
1055 # This branch reads the file into chunks of object arrays and then
1056 # casts them to the desired actual dtype. This ensures correct
1057 # string-length and datetime-unit discovery (like `arr.astype()`).
1058 # Due to chunking, certain error reports are less clear, currently.
1059 if filelike:
TypeError: If a structured dtype is used, the number of columns in `usecols` must match the effective number of fields. But 8 usecols were given and the number of fields is 7.

You can test this dummy dataset
024 xmap_hi.zip

[mikesmic]

.ang files seem to be a mess and not very well defined. I've made a small change on the develop branch that now seems to load your file. You will have to make a small modification to the file though. Please add an empty commented line between NumberFamilies and GRID:, should look like this after:

# NumberFamilies 0
#
# GRID: SqrGrid

Only the first 8 columns of data will be loaded in, as beyond those it seems to a complete guess as to what the data is in all the different .ang formats. Hope this helps, let me know how you get on.

[jorgenasorhaug]

This worked great, thanks! However, I had to change the 'phase' datatype in file_readers.py from uint8 to int8 to read -1 (not indexed): self.data_format = np.dtype([ ... ('phase', 'int8'),])

Further, I see that the reference frames are different between kikuchipy (which I've used to index the patterns) and DefDAP. Is there a way to change the reference frame or rotate my data using DefDAP?

[mikesmic]

Is -1 for non-indexed standard for .ang file? I think the loader currently uses the same convention as the oxford formats, 0 for non-indexed and phases start at 1.

Defdap uses the same convention as oxford ebsd systems for reference frames (spatial frame is x right, y down, z into page and euler angle frame x right, y up, z out). EDAX uses the same convention for spatial frame but the euler frame is rotated 90 deg around z for x down, y right, z out. A correction is applied to the Euler angles when loading .ang files (

DefDAP/defdap/file_readers.py

Line 342 in 4e6d9d6

euler_angle[0] -=np.pi/2

), so you could try changing this to make everything consistent for your input file and we can maybe make the correction a input parameter for the loader.

[jorgenasorhaug]

I'm not sure what the standard of .ang files are, but the python package kikuchipy uses the package orix to make the orientation maps. Orix labels (and saves) the not-indexed pixels as -1.

Thanks for the suggestion. I commented out the line and it seems to be giving me the expected orientation :)

[jorgenasorhaug]

I have issues with reading ctf and ang files saved using Orix and MTEX. I therefore think it would be great to be able to "go around" this by making an EBSD map from the raw data by e.g. reading a numpy file first. However, I'm struggling to figure out how to do so, if possible at all?

[mikesmic]

Your work around would take some effort to implement and I would much prefer to figure out why your files are not loading. Could you send details of the error you get and a small example file that will not load?

[jorgenasorhaug]

I see. Is there then a specific way to write a dictionary file that can read using defdap?

My problems starts here:

File ~\AppData\Local\anaconda3\envs\defdap\Lib\site-packages\defdap\file_readers.py:286, in EdaxAngLoader.load(self, file_name)
283 elif line.startswith('NROWS'):
284 ydim = int(line.split()[-1])
--> 286 shape = (ydim, xdim)
287 self.loaded_metadata['shape'] = shape
289 self.check_metadata()
UnboundLocalError: cannot access local variable 'ydim' where it is not associated with a value

If I manually define these variables in file_readers.py, a different error occurs

File ~\AppData\Local\anaconda3\envs\defdap\Lib\site-packages\numpy\lib\_npyio_impl.py:1046, in _read(fname, delimiter, comment, quote, imaginary_unit, usecols, skiplines, max_rows, converters, ndmin, unpack, dtype, encoding)
1043 data = _preprocess_comments(data, comments, encoding)
1045 if read_dtype_via_object_chunks is None:
-> 1046 arr = _load_from_filelike(
1047 data, delimiter=delimiter, comment=comment, quote=quote,
1048 imaginary_unit=imaginary_unit,
1049 usecols=usecols, skiplines=skiplines, max_rows=max_rows,
1050 converters=converters, dtype=dtype,
1051 encoding=encoding, filelike=filelike,
1052 byte_converters=byte_converters)
1054 else:
1055 # This branch reads the file into chunks of object arrays and then
1056 # casts them to the desired actual dtype. This ensures correct
1057 # string-length and datetime-unit discovery (like `arr.astype()`).
1058 # Due to chunking, certain error reports are less clear, currently.
1059 if filelike:
TypeError: If a structured dtype is used, the number of columns in `usecols` must match the effective number of fields. But 8 usecols were given and the number of fields is 7.

You can test this dummy dataset
024 xmap_hi.zip

[mikesmic]

.ang files seem to be a mess and not very well defined. I've made a small change on the develop branch that now seems to load your file. You will have to make a small modification to the file though. Please add an empty commented line between NumberFamilies and GRID:, should look like this after:

# NumberFamilies 0
#
# GRID: SqrGrid

Only the first 8 columns of data will be loaded in, as beyond those it seems to a complete guess as to what the data is in all the different .ang formats. Hope this helps, let me know how you get on.

[jorgenasorhaug]

This worked great, thanks! However, I had to change the 'phase' datatype in file_readers.py from uint8 to int8 to read -1 (not indexed): self.data_format = np.dtype([ ... ('phase', 'int8'),])

Further, I see that the reference frames are different between kikuchipy (which I've used to index the patterns) and DefDAP. Is there a way to change the reference frame or rotate my data using DefDAP?

[mikesmic]

Is -1 for non-indexed standard for .ang file? I think the loader currently uses the same convention as the oxford formats, 0 for non-indexed and phases start at 1.

Defdap uses the same convention as oxford ebsd systems for reference frames (spatial frame is x right, y down, z into page and euler angle frame x right, y up, z out). EDAX uses the same convention for spatial frame but the euler frame is rotated 90 deg around z for x down, y right, z out. A correction is applied to the Euler angles when loading .ang files (

DefDAP/defdap/file_readers.py

Line 342 in 4e6d9d6

euler_angle[0] -=np.pi/2

), so you could try changing this to make everything consistent for your input file and we can maybe make the correction a input parameter for the loader.

[jorgenasorhaug]

I'm not sure what the standard of .ang files are, but the python package kikuchipy uses the package orix to make the orientation maps. Orix labels (and saves) the not-indexed pixels as -1.

Thanks for the suggestion. I commented out the line and it seems to be giving me the expected orientation :)

[jorgenasorhaug]

I have issues with reading ctf and ang files saved using Orix and MTEX. I therefore think it would be great to be able to "go around" this by making an EBSD map from the raw data by e.g. reading a numpy file first. However, I'm struggling to figure out how to do so, if possible at all?

[mikesmic]

Your work around would take some effort to implement and I would much prefer to figure out why your files are not loading. Could you send details of the error you get and a small example file that will not load?

[jorgenasorhaug]

I see. Is there then a specific way to write a dictionary file that can read using defdap?

My problems starts here:

File ~\AppData\Local\anaconda3\envs\defdap\Lib\site-packages\defdap\file_readers.py:286, in EdaxAngLoader.load(self, file_name)
283 elif line.startswith('NROWS'):
284 ydim = int(line.split()[-1])
--> 286 shape = (ydim, xdim)
287 self.loaded_metadata['shape'] = shape
289 self.check_metadata()
UnboundLocalError: cannot access local variable 'ydim' where it is not associated with a value

If I manually define these variables in file_readers.py, a different error occurs

File ~\AppData\Local\anaconda3\envs\defdap\Lib\site-packages\numpy\lib\_npyio_impl.py:1046, in _read(fname, delimiter, comment, quote, imaginary_unit, usecols, skiplines, max_rows, converters, ndmin, unpack, dtype, encoding)
1043 data = _preprocess_comments(data, comments, encoding)
1045 if read_dtype_via_object_chunks is None:
-> 1046 arr = _load_from_filelike(
1047 data, delimiter=delimiter, comment=comment, quote=quote,
1048 imaginary_unit=imaginary_unit,
1049 usecols=usecols, skiplines=skiplines, max_rows=max_rows,
1050 converters=converters, dtype=dtype,
1051 encoding=encoding, filelike=filelike,
1052 byte_converters=byte_converters)
1054 else:
1055 # This branch reads the file into chunks of object arrays and then
1056 # casts them to the desired actual dtype. This ensures correct
1057 # string-length and datetime-unit discovery (like `arr.astype()`).
1058 # Due to chunking, certain error reports are less clear, currently.
1059 if filelike:
TypeError: If a structured dtype is used, the number of columns in `usecols` must match the effective number of fields. But 8 usecols were given and the number of fields is 7.

You can test this dummy dataset
024 xmap_hi.zip

[mikesmic]

.ang files seem to be a mess and not very well defined. I've made a small change on the develop branch that now seems to load your file. You will have to make a small modification to the file though. Please add an empty commented line between NumberFamilies and GRID:, should look like this after:

# NumberFamilies 0
#
# GRID: SqrGrid

Only the first 8 columns of data will be loaded in, as beyond those it seems to a complete guess as to what the data is in all the different .ang formats. Hope this helps, let me know how you get on.

[jorgenasorhaug]

This worked great, thanks! However, I had to change the 'phase' datatype in file_readers.py from uint8 to int8 to read -1 (not indexed): self.data_format = np.dtype([ ... ('phase', 'int8'),])

Further, I see that the reference frames are different between kikuchipy (which I've used to index the patterns) and DefDAP. Is there a way to change the reference frame or rotate my data using DefDAP?

[mikesmic]

Is -1 for non-indexed standard for .ang file? I think the loader currently uses the same convention as the oxford formats, 0 for non-indexed and phases start at 1.

Defdap uses the same convention as oxford ebsd systems for reference frames (spatial frame is x right, y down, z into page and euler angle frame x right, y up, z out). EDAX uses the same convention for spatial frame but the euler frame is rotated 90 deg around z for x down, y right, z out. A correction is applied to the Euler angles when loading .ang files (

DefDAP/defdap/file_readers.py

Line 342 in 4e6d9d6

euler_angle[0] -=np.pi/2

), so you could try changing this to make everything consistent for your input file and we can maybe make the correction a input parameter for the loader.

[jorgenasorhaug]

I'm not sure what the standard of .ang files are, but the python package kikuchipy uses the package orix to make the orientation maps. Orix labels (and saves) the not-indexed pixels as -1.

Thanks for the suggestion. I commented out the line and it seems to be giving me the expected orientation :)

[jorgenasorhaug]

I have issues with reading ctf and ang files saved using Orix and MTEX. I therefore think it would be great to be able to "go around" this by making an EBSD map from the raw data by e.g. reading a numpy file first. However, I'm struggling to figure out how to do so, if possible at all?

[mikesmic]

Your work around would take some effort to implement and I would much prefer to figure out why your files are not loading. Could you send details of the error you get and a small example file that will not load?

[jorgenasorhaug]

I see. Is there then a specific way to write a dictionary file that can read using defdap?

My problems starts here:

File ~\AppData\Local\anaconda3\envs\defdap\Lib\site-packages\defdap\file_readers.py:286, in EdaxAngLoader.load(self, file_name)
283 elif line.startswith('NROWS'):
284 ydim = int(line.split()[-1])
--> 286 shape = (ydim, xdim)
287 self.loaded_metadata['shape'] = shape
289 self.check_metadata()
UnboundLocalError: cannot access local variable 'ydim' where it is not associated with a value

If I manually define these variables in file_readers.py, a different error occurs

File ~\AppData\Local\anaconda3\envs\defdap\Lib\site-packages\numpy\lib\_npyio_impl.py:1046, in _read(fname, delimiter, comment, quote, imaginary_unit, usecols, skiplines, max_rows, converters, ndmin, unpack, dtype, encoding)
1043 data = _preprocess_comments(data, comments, encoding)
1045 if read_dtype_via_object_chunks is None:
-> 1046 arr = _load_from_filelike(
1047 data, delimiter=delimiter, comment=comment, quote=quote,
1048 imaginary_unit=imaginary_unit,
1049 usecols=usecols, skiplines=skiplines, max_rows=max_rows,
1050 converters=converters, dtype=dtype,
1051 encoding=encoding, filelike=filelike,
1052 byte_converters=byte_converters)
1054 else:
1055 # This branch reads the file into chunks of object arrays and then
1056 # casts them to the desired actual dtype. This ensures correct
1057 # string-length and datetime-unit discovery (like `arr.astype()`).
1058 # Due to chunking, certain error reports are less clear, currently.
1059 if filelike:
TypeError: If a structured dtype is used, the number of columns in `usecols` must match the effective number of fields. But 8 usecols were given and the number of fields is 7.

You can test this dummy dataset
024 xmap_hi.zip

[mikesmic]

.ang files seem to be a mess and not very well defined. I've made a small change on the develop branch that now seems to load your file. You will have to make a small modification to the file though. Please add an empty commented line between NumberFamilies and GRID:, should look like this after:

# NumberFamilies 0
#
# GRID: SqrGrid

Only the first 8 columns of data will be loaded in, as beyond those it seems to a complete guess as to what the data is in all the different .ang formats. Hope this helps, let me know how you get on.

[jorgenasorhaug]

This worked great, thanks! However, I had to change the 'phase' datatype in file_readers.py from uint8 to int8 to read -1 (not indexed): self.data_format = np.dtype([ ... ('phase', 'int8'),])

Further, I see that the reference frames are different between kikuchipy (which I've used to index the patterns) and DefDAP. Is there a way to change the reference frame or rotate my data using DefDAP?

[mikesmic]

Is -1 for non-indexed standard for .ang file? I think the loader currently uses the same convention as the oxford formats, 0 for non-indexed and phases start at 1.

Defdap uses the same convention as oxford ebsd systems for reference frames (spatial frame is x right, y down, z into page and euler angle frame x right, y up, z out). EDAX uses the same convention for spatial frame but the euler frame is rotated 90 deg around z for x down, y right, z out. A correction is applied to the Euler angles when loading .ang files (

DefDAP/defdap/file_readers.py

Line 342 in 4e6d9d6

euler_angle[0] -=np.pi/2

), so you could try changing this to make everything consistent for your input file and we can maybe make the correction a input parameter for the loader.

[jorgenasorhaug]

I'm not sure what the standard of .ang files are, but the python package kikuchipy uses the package orix to make the orientation maps. Orix labels (and saves) the not-indexed pixels as -1.

Thanks for the suggestion. I commented out the line and it seems to be giving me the expected orientation :)