Exploring Solar Eclipses: Data Scraping and Mapping

I saw this post showing a map of the year of the most recent total eclipse, and people mentioning that we can find the data on the Five Millennium Canon of Solar Eclipses Database (the data also mentioned in the thread on ArcGIS don’t go before the seventeenth century).

There is a bit of involved scraping because manually we can’t get more than 4 eclipses in a go and part of the generation is driven by javascript, but we can eventually get the whole 2538 eclipses between -1999 and 2024!

library(tidyverse)
library(rvest)
library(glue)
library(httr)
library(sf)
library(mapview)

eclipses_dates <- read_csv("data/eclipses_dates.csv",
                           col_types = "c") |>
  pull(astro_dates)

aoi <- read_sf("~/data/adminexpress/adminexpress_cog_simpl_000_2022.gpkg",
                   layer = "region") |>
  filter(insee_reg > "06") |>
  st_union() |>
  tibble(geom = _) |>
  st_sf() |>
  mutate(eclipse_date = NA_character_, .before = 1)

get_kmz <- function(eclipse_date) {
  message(glue("Downloading: {eclipse_date}"))
  tryCatch({
    read_html(glue("http://xjubier.free.fr/en/site_pages/solar_eclipses/xSE_GoogleEarth.php?Ecl={eclipse_date}&Acc=2&Umb=1&Lmt=1&Mag=0")) |>
      html_elements("fieldset a") |>
      html_attr("href") |>
      GET(write_disk(glue("results/eclipse_{eclipse_date}.kmz")))
  },
  error = function(e) {
    message("  x Can't download/save")
    print(e)
    return(NULL)
  })
}

eclipses_dates |>
  walk(slowly(get_kmz), .progress = TRUE)

get_polygon <- function(eclipse_date) {
  tryCatch({
    layer <- st_layers(glue("results/eclipse_{eclipse_date}.kmz")) |>
      pluck("name") |>
      keep((x) str_detect(x, "Umbral Path"))

    read_sf(glue("results/eclipse_{eclipse_date}.kmz"),
            layer = layer) |>
      st_zm() |>
      st_make_valid() |>
      st_collection_extract("POLYGON") |>
      transmute(eclipse_date = eclipse_date)
  },
  error = function(e) {
    message("  x Error in opening")
    print(e)
    return(NULL)
  })
}

crop_map <- function(current_map, eclipse_date) {
  message(glue("Date: {eclipse_date}"))
  p <- get_polygon(eclipse_date)
  tryCatch({
    if (any(apply(st_intersects(current_map, p), 1, any))) {
      message("  -> Eclipse matches area of interest")
      bind_rows(st_difference(current_map, p) |>
                  select(eclipse_date),
                st_intersection(current_map, p) |>
                  mutate(eclipse_date = eclipse_date.1) |>
                  select(-eclipse_date.1))
    } else {
      current_map
    }
  },
  error = function(e) {
    message("  x Error in intersection")
    print(e)
    return(current_map)
  })
}

aoi_eclipse <- eclipses_dates |>
  reduce(crop_map, .init = aoi) |>
  write_sf("aoi_eclipse.gpkg")

eclipse_year <- aoi_eclipse |>
  st_make_valid() |>
  group_by(year = str_sub(eclipse_date, 1, 5)) |>
  summarise()

mapview(eclipse_year)

Continue reading: Eclipse map

Analysis of Total Solar Eclipses: Leveraging Data and Mapping Techniques

This article demonstrates how a researcher used various data and mapping strategies to visualise the path of total solar eclipses from years -1999 to 2024. The researcher has used an informative data source known as the Five Millennium Canon of Solar Eclipses Database, and used the programming language R, along with libraries such as tidyverse, rvest, glue, httr, sf, mapview, etc. to process this data.

Long-term Implications

This exercise has long-term implications for not just astrologists, but also for data researchers, environmentalists, educators, programmers, and common public as well. By mapping the path of solar eclipses, one can predict the locations where they can witness this rare phenomenon in future. This can aid the efforts of scientists to gather more data about solar eclipses, and astronomers to forecast future astronomical events with better precision.

Possible Future Developments

In future, this technique can be refined and adapted to other similar phenomena like lunar eclipses or transits of planets. It may also be possible to use more sophisticated data science techniques to extract deeper insights from the data. Some of these might include machine learning algorithms to predict the next locations where solar eclipses will be visible, or the use of more interactive data visualization tools to make the predictions more user-friendly and accessible.

Actionable Advice

Here is some advice on how to leverage these insights:

• For Researchers and Scientists: Utilize the Five Millennium Canon of Solar Eclipses Database and programming languages like R to forecast and analyze solar events, for advancing scientific knowledge.
• For Educators: Use these visualizations to create immersive and practical learning experiences for students studying astronomy.
• For Data Analysts/Scientists: Implement and expand on this technique for other kinds of environmental or spatial data to help drive field-specific innovations.
• For General Public: Use these maps to find out when and where you can observe the next solar eclipse, to enhance your knowledge and fulfill your curiosity about astronomical events.

Read the original article