LiDAR leaks have happened! Black helicopters are in the sky! A few days ago a tiny tweet leaked the online location of „classified LiDAR“ for Washington, DC. This LiDAR really is „classified“ and includes an aerial scan of the Pentagon. For rogue scientists world-wide we offer a secret download link. It links to a file code-named ‚pentagon.laz‚ that contains the 8,044,789 „classified“ returns of the Pentagon shown below. This „classified file“ can be deciphered by any software with native LAZ support. It was encrypted with the „LAS 1.4 compatibility mode“ of LASzip. The original LAS 1.4 content was encoded into a inconspicuous-looking LAZ file. New point attributes (such as the scanner channel) were hidden as „extra bytes“ for fully lossless encryption. Use ‚laszip‚ to fully decode the original „classified“ LAS 1.4 file … (-;
Seriously, a tiled LiDAR data set for the District of Columbia flown in 2015 is available for anyone to use on Amazon S3 with a very permissive open data license, namely the Creative Commons Attribution 3.0 License. The LiDAR coverage can be explored via this interactive map. The tiles are provided in LAS 1.4 format and use the new point type 6. We downloaded a few tiles near the White House, the Capitol, and the Pentagon to test the „native LAS 1.4 extension“ of our LASzip compressor which will be released soon (a prototype for testing is already available). As these uncompressed LAS files are YUUUGE we use the command line utility ‚wget‚ for downloading. With option ‚-c‘ the download continues where it left off in case the transfer gets interrupted.
We use lasboundary to create labeled bounding boxes for display in Google Earth and lasgrid to a create false color visualization of pulse density with the command lines shown below. Pulse densities of 20 or below are mapped to blue. Pulse densities of 100 or above are mapped to red. We picked the min value 20 and the max value 100 for this false color mapping by running lasinfo with the ‚-cd‘ option to compute an average pulse density and then refining the numbers experimentally. We also use lasoverlap to visualize how flightlines overlap and how well they align. Vertical differences of up to 20 cm are mapped to white and differences of 40 cm or more are mapped to saturated blue or red.
lasboundary -i *.las ^
-use_bb ^
-labels ^
-odir quality -odix _bb -okml
lasgrid -i *.las ^
-keep_last ^
-point_density -step 2 ^
-false -set_min_max 20 100 ^
-odir quality -odix _d_20_100 -opng ^
-cores 2
lasoverlap -i *.las ^
-min_diff 0.2 -max_diff 0.4 ^
-odir quality -opng ^
-cores 2
The visualization of the pulse density and of the flightline overlap both show that there is no LiDAR for the White House or Capitol Hill. We will never know how tall the tomato and kale plants had grown in Michelle Obama’s organic garden on that day. Note that the White House and Capitol Hill were not simply „cut out“. Instead the flight plan of the survey plane was carefully designed to avoid these areas. Surprisingly, the Pentagon did not receive the same treatment and is (almost) fully included in the open LiDAR as mentioned in the dramatic first paragraph. Interesting is how the varying (tidal?) water level of the Potomac River shows up in the visualization of flightline miss-alignments.
There are a number of issues in these LiDAR files. The most serious ones are reported at the very end of this article. We will now scrutinize the partly-filled tile 2016.las close to the White House with only 11,060,334 returns. A lasvalidate check immediately reports three deviations from the LAS 1.4 specification:
lasvalidate -i 2016.las -o 2016_check.xml
- For proper LAS 1.4 files containing point type 6 through 10 all ‚legacy‘ point counts in the LAS header should be set to 0. The following six fields in the LAS header should be zero for tile 2016.las (and all other tiles):
+ legacy number of point records
+ legacy number of points by return[0]
+ legacy number of points by return[1]
+ legacy number of points by return[2]
+ legacy number of points by return[3]
+ legacy number of points by return[4] - There should not be any LiDAR return in a valid LAS file whose ’number of returns of given pulse‘ attribute is zero but there are 8 such points in tile 2016.las (and many more in various other tiles).
- There should not be any LiDAR return whose ‚return number‘ attribute is larger than their ’number of returns of given pulse‘ attribute but there are 8 such points in tile 2016.las (and many more in various other tiles).
The first issue is trivial. There is an efficient in-place fix that does not require to rewrite the entire file using lasinfo with the following command line:
lasinfo -i 2016.las ^
-nh -nv -nc ^
-set_number_of_point_records 0 ^
-set_number_of_points_by_return 0 0 0 0 0 ^
A quick check with las2txt shows us that the second and third issue are caused by the same eight points. Instead of writing an 8 for the ’number of returns‘ attribute the LAS file exporter must have written a 0 (marked in red for all eight returns) and instead of writing an 8 for the ‚return number‘ attribute the LAS file exporter must have written a 1 (also marked in red). We can tell it from the true first return via its z coordinate (marked in blue) as the last return should be the lowest of all.
las2txt -i 2016.las ^
-keep_number_of_returns 0 ^
-parse xyzrnt ^
-stdout
397372.70 136671.62 33.02 4 0 112813299.954811
397372.03 136671.64 28.50 5 0 112813299.954811
397371.28 136671.67 23.48 6 0 112813299.954811
397370.30 136671.68 16.86 7 0 112813299.954811
397369.65 136671.70 12.50 1 0 112813299.954811
397374.37 136671.58 44.17 3 0 112813299.954811
397375.46 136671.56 51.49 1 0 112813299.954811
397374.86 136671.57 47.45 2 0 112813299.954811
With las2las we can change the ’number of returns‘ from 0 to 8 using a ‚-filtered_transform‘ as illustrated in the command line below. We suspect that higher number of returns such as 9 or 10 might have been mapped to 1 and 2. Fixing those as well as repairing the wrong return numbers will require a more complex tool. We would recommend to check all tiles with more scrutiny using the lasreturn tool. But wait … more return numbering issues are to come.
las2las -i 2016.las ^
-keep_number_of_returns 0 ^
-filtered_transform ^
-set_extended_number_of_returns 8 ^
-odix _fixed -olas
A closer look at the scan pattern reveals that the LiDAR survey was flown with a dual-beam system where two laser beams scan the terrain simultaneously. This is evident in the textual representation below as there are multiple „sets of returns“ for the same GPS time stamp such as 112813952.110394. We group the returns from the two beams into an orange and a green group. Their coordinates show that the two laser beams point into different directions when they are simultaneously „shot“ and therefore hit the terrain far apart from another.
las2txt -i 2016.las ^
-keep_gps_time 112813952.110392 112813952.110396 ^
-parse xyzlurntp ^
-stdout
397271.40 136832.35 54.31 0 0 1 1 112813952.110394 117
397277.36 136793.35 38.68 0 1 1 4 112813952.110394 117
397277.35 136793.56 32.89 0 1 2 4 112813952.110394 117
397277.34 136793.88 24.13 0 1 3 4 112813952.110394 117
397277.32 136794.25 13.66 0 1 4 4 112813952.110394 117
The information about which point is from which beam is currently stored into the generic ‚user data‘ attribute instead of into the dedicated ’scanner channel‘ attribute. This can be fixed with las2las as follows.
las2las -i 2016.las ^
-copy_user_data_into_scanner_channel ^
-set_user_data 0 ^
-odix _fixed -olas
Unfortunately the LiDAR files have much more serious issues in the return numbering. It’s literally a „Total Disaster!“ and „Sad!“ as the US president will tweet shortly. After grouping all returns with the same GPS time stamp into an orange and a green group there is one more set of returns left unaccounted for.
las2txt -i 2016.las ^
-keep_gps_time 112813951.416451 112813951.416455 ^
-parse xyzlurntpi ^
-stdout
397286.02 136790.60 45.90 0 0 1 4 112813951.416453 117 24
397286.06 136791.05 39.54 0 0 2 4 112813951.416453 117 35
397286.10 136791.51 33.34 0 0 3 4 112813951.416453 117 24
397286.18 136792.41 21.11 0 0 4 4 112813951.416453 117 0
397286.12 136791.75 30.07 0 0 1 1 112813951.416453 117 47
397291.74 136750.70 45.86 0 1 1 1 112813951.416453 117 105
las2txt -i 2016.las ^
-keep_gps_time 112813951.408708 112813951.408712 ^
-parse xyzlurntpi ^
-stdout
397286.01 136790.06 45.84 0 0 1 4 112813951.408710 117 7
397286.05 136790.51 39.56 0 0 2 4 112813951.408710 117 15
397286.08 136790.96 33.33 0 0 3 4 112813951.408710 117 19
397286.18 136792.16 17.05 0 0 4 4 112813951.408710 117 0
397286.11 136791.20 30.03 0 0 1 2 112813951.408710 117 58
397286.14 136791.67 23.81 0 0 2 2 112813951.408710 117 42
397291.73 136750.16 45.88 0 1 1 1 112813951.408710 117 142
This can be visualized with lasview and the result is unmistakably clear: The return numbering is messed up. There should be one shot with five returns (not a group of four and a single return) in the first example. And there should be one shot with six returns (not a group of four and a group of two returns) in the second example. Such a broken return numbering results in extra first (or last) returns. These are serious issues that affect any algorithm that relies on the return numbering such as first-return DSM generation or canopy cover computation. Those extra returns will also make the pulse density appear higher and the pulse spacing appear tighter than they really are. The numbers from 20 (blue) to 100 (red) pulses per square meters in our earlier visualization are definitely inflated.
lasview -i 2016.las ^
-keep_gps_time 112813951.416451 112813951.416455 ^
-color_by_return
lasview -i 2016.las ^
-keep_gps_time 112813951.408708 112813951.408712 ^
-color_by_return
After all these troubles here something nice. Side-by-side a first-return TIN and a spike-free TIN (using a freeze of 0.8 m) of the center court cafe in the Pentagon. Especially given all these „fake first returns“ in the Washington DC LiDAR we really need the spike-free algorithm to finally „Make a DSM great again!“ … (-;
We would like to acknowledge the District of Columbia Office of the Chief Technology Officer (OCTO) for providing this data with a very permissive open data license, namely the Creative Commons Attribution 3.0 License.