When you do this in an Excel pivot table, Excel will indeed compute the results for all rows. The trick is to create a measure that will only return results for the top 10.

Start with a measure that calculates the marks by student. As you cannot use a column directly, it has to be aggregated. I would assume the average mark is a good candidate for this:

AvgMark:=AVERAGE(Student[Marks])

The TOPN function returns the top rows, based on a measure. In our case, this will be [AvgMark]:

AvgMarkTop10:=CALCULATE([AvgMark],TOPN(10,Student,[AvgMark]))

When you use [AvgMarkTop10] in a pivot table with Student Name as row labels, for each row the context is changed from the selected student to the top 10 students. In other words, every row will show the average marks for the top 10. This is obviously not
what you want. The last step needed is to reapply the original context as a filter, with the VALUES function. By doing this, the context will become the selected student if this student is in the top 10, and empty if the student is not in the top 10:

MarksTop10Only:=CALCULATE([AvgMark],TOPN(10, Student, [AvgMark]),VALUES(Student[Student Name]))