Reproducibility for NZ coast

[neon-ninja]

Hi @kvos,

Thanks for making this open source. I'm trying to partially update the dataset published at https://zenodo.org/records/7758183, to add in shorelines from satellite imagery from 2022-2024 in New Zealand. One of the problems I noticed while trying to do this, is that I don't get the same results as you (pre-tidal correction), even for the exact same satellite imagery.

I'm using @2320sharon's coastsat-package (https://github.com/SatelliteShorelines/coastsat_package, 0.1.73)

Here's my notebook: https://github.com/UoA-eResearch/CoastSat/blob/b3a9739af2e26fae26ed4b7bf68dea8b040ed9e0/update.ipynb

I'm looking at site nzd0188, which is Tairua beach

I'm using the polygons.geojson, shorelines.geojson, and transects.geojson from that Zenodo link

I noticed a problem with the reference shoreline though, it didn't have enough points to accurately build a mask, as that relies on buffering from the points that construe the line. So I had to split the shoreline by the transects to add in enough points.

I've also noticed I've had to flip coordinate order (lat lng -> lng lat) at times in order for things to work

As you can see from the table at the bottom of my notebook, my calculated intersects are typically off from yours by roughly 40m

Any ideas what I'm doing wrong?

Cheers,
Nick

[kvos]

hi nick, which version of CoastSat are you using? I would need to dig out the parameters that I used for NZ (the different settings + classifier used can make a big difference). Tidal correction can also contribute to it, although 40 m bias is huge.

[neon-ninja]

I've tried with https://pypi.org/project/coastsat-package/0.1.73/ and https://github.com/kvos/CoastSat/releases/tag/v2.6 and had the same problem. I'm using these settings:

# settings for the shoreline extraction
settings = {
    # general parameters:
    'cloud_thresh': 0.1,        # threshold on maximum cloud cover
    'dist_clouds': 100,         # ditance around clouds where shoreline can't be mapped
    'output_epsg': 2193,       # epsg code of spatial reference system desired for the output
    # quality control:
    'check_detection': False,    # if True, shows each shoreline detection to the user for validation
    'adjust_detection': False,  # if True, allows user to adjust the postion of each shoreline by changing the threhold
    'save_figure': True,        # if True, saves a figure showing the mapped shoreline for each image
    # [ONLY FOR ADVANCED USERS] shoreline detection parameters:
    'min_beach_area': 1000,     # minimum area (in metres^2) for an object to be labelled as a beach
    'min_length_sl': 500,       # minimum length (in metres) of shoreline perimeter to be valid
    'cloud_mask_issue': False,  # switch this parameter to True if sand pixels are masked (in black) on many images
    'sand_color': 'default',    # 'default', 'latest', 'dark' (for grey/black sand beaches) or 'bright' (for white sand beaches)
    'pan_off': False,           # True to switch pansharpening off for Landsat 7/8/9 imagery
    's2cloudless_prob': 40,      # threshold to identify cloud pixels in the s2cloudless probability mask
    # add the inputs defined previously
    'inputs': inputs
}

I did also try with 'sand_color': 'latest' but had the same problem

[2320sharon]

Hi Nick,
Thanks for raising this issue. A difference of 40m between your data and the original data is significant. Without knowing what the original settings that were used to get the original results its difficult to conclude whether this is an issue with coastsat_package or a difference in results due to different settings.

Its possible that your reference shoreline buffer is too small if the reference shoreline buffer isn't covering enough of your beach causing you to have to resample the reference shoreline. You could try setting max_dist_ref to a higher value to dilate the buffer. I see in your script its set to a default of 100 meters settings["max_dist_ref"] = 100. I would recommend increasing this to ensure your entire beach is being covered. If you increase this value its possible the original reference shoreline will work even if it consists of only a few points, but if that doesn't work I would keep using your resampled reference shoreline.

Coastsat_package is meant to replicate the original coastsat with only some minor alterations that are documented in the readme. I'll take a look at the code to see if there is anything wrong that could be causing this issue, but my hunch is that this difference is due to different settings. You could try experimenting by changing the 'min_beach_area' and 'min_length_sl' to see if that changes the results.

[neon-ninja]

I don't think it's a problem with coastsat_package, as I tried with both coastsat_package and this repo directly and observed the same problem. The coordinate flip issue seems to only occur with coastsat_package, but that's easily fixed with np.flip. I tried increasing max_dist_ref to 300, here's the resulting plot:

Seems a bit noisier than 100:

The offset issue still occurs with settings["max_dist_ref"] = 300.

I tried halving min_beach_area and min_length_sl, same issue

[neon-ninja]

Could this be a CRS issue? What CRS did you use @kvos?

[kvos]

as long as you have used a local projection system CRS should be fine (only small differences).
Is this with Landsat? Note that the data on the coastsat website is only Landsat Collection 1 with CoastSat v1, while now you are using Landsat Collection 2 with CoastSat v2.
This are the settings that I used when I processed the NZ data.
New_Zealand_settings_to_map_shorelines_v5.txt

Maybe try a different site to see if the differences are also so large. CoastSat v1 and v2 give very similar results in most places but occasionally due to a pixel rounding issue with GEE there is a pixel shift between L5 and L7/L8, so that was the main fix from v1 to v2.

[neon-ninja]

Yes - I'm using L5, L7 and L8. We seem to have a good overlap in terms of satellite imagery timestamps, judging by this venn diagram:

I tried with those settings, same problem

I've been downloading imagery in parallel for all of the NZ sites over the past few days (with https://github.com/UoA-eResearch/CoastSat/blob/main/batch_process.py), so am in a position to perform large scale comparison. For the 457 sites I've processed so far, comparing with your results for those exact sites, here's the output of .describe():

count    457.000000
mean     -40.086508
std       19.911214
min     -101.793516
25%      -52.151887
50%      -38.624932
75%      -27.483396
max       86.271906
Name: diff, dtype: float64

or, looking at the composite transects:

count    24573.000000
mean       -42.372753
std         20.819492
min       -332.331356
25%        -56.321811
50%        -41.023628
75%        -28.938976
max        181.798978
dtype: float64

As a histogram:

[neon-ninja]

I've also run into a separate issue with transects.geojson (https://zenodo.org/records/7758183/files/transects.geojson?download=1) - the following ids are duplicated:

	id	count
6	nzd0418-0000	2
5	nzd0418-0001	2
4	nzd0418-0002	2
3	nzd0418-0003	2
2	nzd0418-0004	2
11	nzd0418-0005	2
21	nzd0418-0006	2
7	nzd0419-0000	2
8	nzd0419-0001	2
9	nzd0419-0002	2
10	nzd0419-0003	2
12	nzd0419-0004	2
1	nzd0419-0005	2
13	nzd0419-0006	2
0	nzd0419-0007	2
14	nzd0419-0008	2
15	nzd0419-0009	2
16	nzd0419-0010	2
17	nzd0419-0011	2
18	nzd0419-0012	2
19	nzd0419-0013	2
20	nzd0419-0014	2

[neon-ninja]

I've now downloaded all 560 sites, updated stats and plot:

count    32004.000000
mean       -44.427866
std         21.820599
min       -357.977268
25%        -59.491487
50%        -42.821800
75%        -30.240118
max        181.798978
dtype: float64

[kvos]

I cannot tell what is happening by looking at the distribution of differences, but if you plot a few transects with the time-series from both coastsat-website and your run I may be able to pick some differences. If you can color/symbol each sensor that would also help.

[neon-ninja]

Sure - here's a plot of results over time for just one transect (nzd0202-0002):

At least the trends match

Did you perform any normalisation / mean adjustment to time_series_raw.csv?

[neon-ninja]

Here's a plot for 2003-07-21-21-42-33_L5_nzd0188_ms.tif

And here's the same imagery, with transects, the reference shoreline, my calculated intersects along those transects in red, and yours in blue:

[neon-ninja]

Could this be a collection 1 vs 2 issue?

[kvos]

Did you perform any normalisation / mean adjustment to time_series_raw.csv?

ok that's a big difference. time_series_raw.csv are not tidally corrected. Collection 1 vs Collection 2 would not create such differences as it's only some improvement of georeferencing of the images.

[kvos]

Here's a plot for 2003-07-21-21-42-33_L5_nzd0188_ms.tif

ok but you are using coastseg not coastsat. Try with coastsat so at least we can discard on possibility.