Exercise 4#
Deadline:Please complete this exercise by the end of day on Thursday, 28 November, 2024 (the day of next week’s work session).
To start this assignment, accept the GitHub Classroom assignment, and clone your own repository, e.g., in a CSC Noppe instance. Make sure you commit and push all changes you make (you can revisit instructions on how to use git
and the jupyterlab git-plugin on the website of the Geo-Python course.
To preview the exercise without logging in, you can find the open course copy of the course’s GitHub repository at github.com/automating-gis-processes-2024/exercise-4. Don’t attempt to commit changes to that repository, but rather work with your personal GitHub Classroom copy (see above).
Hints#
Joining two data frames on different column names#
We have already joined data sets that share the same index, and also used spatial joins to merge geo-data frames depending on their geometric relationships.
For problem 1, it might be handy to be able to join two data sets using the values of two columns that have a different name. One good approach is to set the index of both data frames to refer to the same column:
[ ]:
import pandas
df1 = pandas.DataFrame({
"id": [1, 2, 3],
"other_column": ["a", "b", "c"]
})
df1
[ ]:
df2 = pandas.DataFrame({
"id": [67, 68, 69],
"other_other_column": ["x", "y", "z"],
"df1_id": [1, 2, 3]
})
df2
[ ]:
joint_df = df1.set_index("id").join(df2.set_index("df1_id"))
joint_df
Renaming columns when joining data frames#
It is often necessary to rename columns when we join data frames that have duplicate column names. In the example below, both df1
and df2
have a column other_column
; the join fails. An appropriate fix is to add a suffix to all columns of one or both of the data frames:
[ ]:
import pandas
df1 = pandas.DataFrame({
"id": [1, 2, 3],
"other_column": ["a", "b", "c"]
})
df1
[ ]:
df2 = pandas.DataFrame({
"id": [67, 68, 69],
"other_other_column": ["x", "y", "z"],
"df1_id": [1, 2, 3]
})
df2
[ ]:
# Will fail, because duplicate column names exist:
joint_df = df1.join(df2)
joint_df
[ ]:
# works: add a suffix to one of the data sets’ columns
joint_df = df1.join(df2.add_suffix("_df2"))
joint_df
Searching for files using a pattern#
In Lesson 2 we discussed how to use a file pattern to search for files, using `pathlib.Path.glob()
<https://docs.python.org/3/library/pathlib.html#pathlib.Path.glob>`__.
To loop over all files ending in _s.shp
inside DATA_DIRECTORY / "finland_topographic_database
, use the following piece of code:
[ ]:
import pathlib
DATA_DIRECTORY = pathlib.Path().resolve() / "data"
for input_file in (DATA_DIRECTORY / "finland_topographic_database").glob("*_s.shp"):
print(input_file.name)
This will come in handy for problem 2, when reading in all travel time data files. Be sure to revisit the explanation in Lesson 2.
Find the minimum value across multiple columns#
For problem 2, you have to find the smallest value across multiple columns: the shortest travel time to any of the eight shopping centres. For this, `panda
’s DataFrame.min()
method <https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.min.html>`__ can come in handy. It identifies the smallest value in each column or row (it accepts the same axis=
parameter as apply()
).
For instance, to find the smalles value for each row across the columns a
, b
, and c
of the data frame below, use the following code:
[ ]:
import pandas
df = pandas.DataFrame(
{
"id": [1, 2, 3],
"a": [27, 64, 12],
"b": [13, 13, 13],
"c": [34, 15, 1]
}
)
df
[ ]:
# select which columns to compare, then call `.min()`
df[["a", "b", "c"]].min(axis=1)
To find out which column had the smallest value for each row, use the near-identical method `idxmin()
<https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.idxmin.html>`__:
[ ]:
df[["a", "b", "c"]].idxmin(axis=1)
Of course, equivalent methods to find the greatest values exist: they are named `pandas.DataFrame.max()
<https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.max.html>`__ and `pandas.DataFrame.idxmax()
<https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.idxmax.html>`__.
Custom classification with mapclassify
#
mapclassify.UserDefined
allows grouping data into custom classes based on predefined breakpoints. This method is ideal for thematic mapping and tailoring classifications to specific ranges.
yb: Short for “yielded bins,” this attribute contains the class index for each data value
[1]:
import geopandas as gpd
from mapclassify import UserDefined
# Sample data
data = {"region": ["A", "B", "C", "D", "E"], "population_density": [120, 450, 800, 300, 1500]}
gdf = gpd.GeoDataFrame(data)
# Define custom breakpoints
breaks = [200, 500, 1000, 2000]
# Classify data with UserDefined
classifier = UserDefined(gdf["population_density"], bins=breaks)
gdf["class"] = classifier.yb # Assign class indices to a new column