The following don't answer the question of why they came back, but at least we know where they went.
New York Passenger Arrival Lists (Ellis Island), 1892-1924
Name: Johanna Olsson
Event Type: Immigration
Event Date: 07 Apr 1903
Event Place: Ellis Island, New York City, New York, United States
Event Place (Original): Ellis Island, New York City, New York, United States
Residence Place: Ostraby, Sweden
Sex: Female
Age: 24
Marital Status: Single
Nationality: Sweden, Swedish
Birth Year: 1879
Departure Port: Liverpool
Ship Name: Cymric
Second Additional Person's Name: Mr O J Olssen
Additional Name 2 Relationship: Brother
Page Number: 24
Affiliate Line Number: 0026
She's travelling with brother August and they are going to their brother Mr. O.J. Olson in Dawson, Minnesota.
New York Passenger Arrival Lists (Ellis Island), 1892-1924
Name: Oln Persson [mispelled by indexer]
Event Type: Immigration
Event Date: 02 Apr 1904
Event Place: Ellis Island, New York City, New York, United States
Event Place (Original): Ellis Island, New York City, New York, United States
Residence Place: Ostraby
Sex: Male
Age: 62
Marital Status: Married
Nationality: Sweden, Swedish
Birth Year: 1842
Departure Port: Copenhagen
Ship Name: United States
Second Additional Person's Name: Ole Y Olson
Additional Name 2 Relationship: Son
Page Number: 38
Affiliate Line Number: 0014
He's travelling with daughter Anna and Anna's son Anders, they are also going to his son Ole J. Olson in Dawson Minnesota. I think you probably know already that his son Ola was born 12 Apr 1873 and this is probably the son Ola with sister Johanna in the 1910 US census. Their brother August also is on the following page but I guess the indexer didn't realize it was the same household.
United States Census, 1910
Name: Ole Olson
Event Type: Census
Event Date: 1910
Event Place: Maxwell Township, Lac qui Parle, Minnesota, United States
Event Place (Original): Maxwell, 72, Lac Qui Parle, Minnesota
Sex: Male
Age: 37
Marital Status: Single
Race: White
Race (Original): White
Birth Year (Estimated): 1873
Birthplace: Sweden
Immigration Year: 1901
Father's Birthplace: Sweden
Mother's Birthplace: Sweden
Relationship to Head of Household: Head
Relationship to Head of Household (Original): Head
Sheet Letter: B
Sheet Number: 15
Household Role Sex Age Birthplace
Ole Olson Head Male 37 Sweden
Johannah Olson Sister Female 29 Sweden