Forum Discussion

Altera_Forum's avatar
Altera_Forum
Icon for Honored Contributor rankHonored Contributor
9 years ago

How does the kernel determine the resource flags of a device node in a device tree?

I'm developing a devicedriver in a socfpga-linux provided by SoC EDS 15.1.1.60 for a customFPGA based PCIe Root Complex design on Altera Arria 10 board. I'mtaking pcie-altera.c for reference. When my driver parses the devicetree for the "ranges" property, it fails. Let me give outthe sequence:

drivers/pci/host/pcie-altera.c:

altera_pcie_probe() ->altera_pcie_parse_request_of_pci_ranges().

In altera_pcie_parse_request_of_pci_ranges(), the foll. piece of codedecides checks the device node flags, and decides whether the devicenode (in device tree) is prefetchable or not:

resource_list_for_each_entry(win, &pcie->resources) {

structresource *parent, *res = win->res;

switch(resource_type(res)) {

caseIORESOURCE_MEM:

parent = &iomem_resource;

res_valid |= !(res->flags & IORESOURCE_PREFETCH);

break;

default:

continue;

}

err =devm_request_resource(dev, parent, res);

if (err)

gotoout_release_res;

}

if (!res_valid) {

dev_err(dev,"non-prefetchable memory resource required\n");

err =-EINVAL;

gotoout_release_res;

}

In my case, the device nodeflag happens to be IORESOURCE_BUS, so it falls into the "default"case of switch. Since the res_valid variable is initialized to zeroin the beginning, my code falls into "if(!res-valid)"block and exits with error.

My DTS file is similar toaltera's/rocketboard's pcie DTS file. I've given below a part of myDTS file for reference:

sopc0: sopc@0 {

device_type= "soc";

ranges;

# address-cells = <1>;

# size-cells= <1>;

compatible ="ALTR,avalon", "simple-bus";

bus-frequency = <0>;

test_subsys_pcie: pcie@0x000000000 {

compatible = "altera, my-pcie";

reg= <0xc0000000 0x00001000>,

<0xc0001000 0x00010000>,

<0xff200000 0x00010000>;

reg-names = "axi_slave_1", "axi_slave_2","axi_slave_3";

interrupt-parent = <&HPS_arm_gic_0>;

interrupts = <0 19 4>;

clocks = <&test_subsys_clk_125M>;

device_type = "pci";

bus-range = <0x00000000 0x000000ff>;

# address-cells = <1>;

# size-cells = <1>;

ranges = <0x00000000 0xc0000000 0x00001000>;

}; //endunknown@0x000000000 (test_subsys_pcie)

Partof Rocketboard's DTS file is given below for reference:

sopc0:sopc@0 {

device_type= "soc";

ranges;

# address-cells= <1>;

# size-cells= <1>;

compatible= "ALTR,avalon", "simple-bus";

bus-frequency= <0>;

pcie_0_pcie_a10_hip_avmm:pcie@0x010000000 {

compatible= "altr,pcie-root-port-15.1", "altr,pcie-root-port-1.0";

reg= <0xd0000000 0x10000000>,

<0xff2100000x00004000>;

reg-names= "Txs", "Cra";

interrupt-parent= <&arria10_hps_0_arm_gic_0>;

interrupts= <0 24 4>;

interrupt-controller;

# interrupt-cells= <1>;

device_type= "pci"; /* embeddedsw.dts.params.device_type type STRING*/

msi-parent= <&pcie_0_msi_to_gic_gen_0>;

bus-range= <0x00000000 0x000000ff>;

# address-cells= <3>;

# size-cells= <2>;

ranges= <0x82000000 0x00000000 0x00000000 0xd0000000 0x000000000x10000000>;

interrupt-map-mask= <0 0 0 7>;

interrupt-map= <0 0 0 1 &pcie_0_pcie_a10_hip_avmm 1>,

<00 0 2 &pcie_0_pcie_a10_hip_avmm 2>,

<00 0 3 &pcie_0_pcie_a10_hip_avmm 3>,

<00 0 4 &pcie_0_pcie_a10_hip_avmm 4>;

};//end pcie@0x010000000 (pcie_0_pcie_a10_hip_avmm)

Both in my DTS file androcketboard's DTS file, the flags of the device node have NOT beenmentioned. Since the rocketboard's pcie-altera.c file has ONLY the"case IORESOURCE_MEM" in switch..., it is obvious thattheir device node seems to be of IORESOURCE_MEM.

I've 4 questions here.

1) How does therocketboard's device node is taken to be of IORESOURCE_MEM type inspite of their DTS file not mentioning the device node flag anywhere?

2) How does my device nodeis taken to be of IORESOURCE_BUS type, inspite of me following thesame format as rocketboard in DTS as well as code?

3) Is there actually any wayto specify the device node flags in DTS file?

4) When the kernel parsesthe DTS file, how does it determine the flags of a particular devicenode?

Please help me out.

2 Replies

  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    9 years ago

    I further narrowed down the issue.

    My top-level structure is this:

    struct test_pcie {

    struct platform_device *pdev;

    void __iomem *axi_slave_base_1;

    void __iomem *axi_slave_base_2;

    void __iomem *axi_slave_base_3;

    int irq;

    u8 root_bus_nr;

    struct irq_domain *irq_domain;

    struct resource bus_range;

    struct list_head resources;

    };

    When I register the driver, I printed out what resources i.e. list of 'struct resource' it has, and it's types. I wrote a dump() function which lists out the resources.

    void dump_resources (struct platform_device *dev)

    {

    int i;

    for (i = 0; i < dev->num_resources; i++) {

    struct resource *r = &dev->resource[i];

    if (r->name != NULL)

    printk("%s: r->name=%s, r->type=%lx\n", __FUNCTION__, r->name, resource_type(r));

    }

    }

    I passed "pointer_to_struct_pcie->pdev" to dump_resources and this is what I got:

    [ 17.078754] dump_resources: r->name=axi_slave_1, r->type=0x200

    [ 17.084833] dump_resources: r->name=axi_slave_2, r->type=0x200

    [ 17.090722] dump_resources: r->name=axi_slave_3, r->type=0x200

    [ 17.096531] dump_resources: r->name=/sopc@0/pcie@0x000000000, r->type=0x400

    So, in the beginning, the driver has 4 resources, out of which 3 are of IORESOURCE_MEM type (r->type=0x200), and 1 is of IORESOURCE_BUS type (r->type=0x400).

    The 3 IORESOURCE_MEM type resources i.e. axi_slave_1, axi_slave_2, axi_slave_3 are listed under the device node of "/sopc@0/pcie@0x000000000" (IORESOURCE_BUS type) as can be evidenced from the DTS file.

    The problem comes when I try to parse the ranges inside the "/sopc@0/pcie@0x000000000" device node, with the call to altera_pcie_parse_request_of_pci_ranges() which calls of_pci_get_host_bridge_resources(np, 0, 0xff, &pcie->resources, NULL), and then loops each of the pcie->resources obtained until it hits a IORESOURCE_MEM type resource. In the loop, I printed out each resource name in the list, but I found that it has ONLY one resource in the list:

    resource_list_for_each_entry(win, &pcie->resources) {

    struct resource *parent, *res = win->res;

    printk("resource: name=%s, flags=0x%lx\n", res->name, res->flags);

    switch (resource_type(res)) {

    case IORESOURCE_MEM:

    parent = &iomem_resource;

    res_valid |= !(res->flags & IORESOURCE_PREFETCH);

    printk("res_valid = %d\n", res_valid);

    break;

    default:

    continue;

    }

    if (!res_valid) {

    dev_err(dev, "non-prefetchable memory resource required\n");

    err = -EINVAL;

    goto out_release_res;

    }

    The output I got was:

    [ 17.183229] resource: name=pcie, flags=0x1000

    This resource "pcie" is nothing but the device node that encapsulates the sub-resources under it viz. axi_slave_1, axi_slave_2, axi_slave_3. The flags 0x1000 show the "pcie" device node as IORESOURCE_BUS type, so it doesn't fall into "case IORESOURCE_MEM" but rather into "case default" and exits. This clearly shows that axi_slave_1, axi_slave_2, axi_slave_3 are not part of "pcie->resources" list. While the device node seems to have been mapped, the sub-resources under it seem to have been not mapped.

    So, I feel I need to dig into of_pci_get_host_bridge_resources(np, 0, 0xff, &pcie->resources, NULL) and check why pcie->resources contains only one resource i.e the top-level "pcie" device node, but not the sub-resources under it. But in the beginning, when I register the driver, I could see the top-level resource as well as it's sub-resources, as given in the prints above.

    If someone has got clue already as to what might have gone wrong or what is needed to be done, please answer.
  • Altera_Forum's avatar
    Altera_Forum
    Icon for Honored Contributor rankHonored Contributor
    9 years ago

    On further debugging, I observed that the parsing of "ranges' property fails at this point:

    To give the sequence:

    altera_pcie_probe() -> altera_pcie_parse_request_of_pci_ranges() -> of_pci_get_host_bridge_resources()

    In of_pci_get_host_bridge_resources(), there is a call to of_pci_range_parser_init(), and then a loop for_each_of_pci_range(&parser, &range), as below:

    /* Check for ranges property */

    err = of_pci_range_parser_init(&parser, dev);

    if (err)

    goto parse_failed;

    pr_debug("Parsing ranges property...\n");

    for_each_of_pci_range(&parser, &range) {

    pr_info("inside for_each_of_pci_range...\n");

    /* Read next ranges element */

    if ((range.flags & IORESOURCE_TYPE_BITS) == IORESOURCE_IO)

    snprintf(range_type, 4, " IO");

    else if ((range.flags & IORESOURCE_TYPE_BITS) == IORESOURCE_MEM)

    snprintf(range_type, 4, "MEM");

    else

    snprintf(range_type, 4, "err");

    pr_info(" %s %#010llx..%#010llx -> %#010llx\n", range_type,

    range.cpu_addr, range.cpu_addr + range.size - 1,

    range.pci_addr);

    /* some code goes here */

    }

    What I observed is, the loop is not entered at all, which means the previous function of_pci_range_parser_init() hasn't done it's job properly.

    Let me paste a portion of my DTS file again below:

    sopc0: sopc@0 {

    device_type = "soc";

    ranges;

    # address-cells = <1>;

    # size-cells = <1>;

    compatible = "ALTR,avalon", "simple-bus";

    bus-frequency = <0>;

    test_subsys_pcie: pcie@0x000000000 {

    compatible = "altera, my-pcie";

    reg = <0xc0000000 0x00001000>,

    <0xc0001000 0x00010000>,

    <0xff200000 0x00010000>;

    reg-names = "axi_slave_1", "axi_slave_2", "axi_slave_3";

    interrupt-parent = <&HPS_arm_gic_0>;

    interrupts = <0 19 4>;

    clocks = <&test_subsys_clk_125M>;

    device_type = "pci";

    bus-range = <0x00000000 0x000000ff>;

    # address-cells = <1>;

    # size-cells = <1>;

    ranges = <0x00000000 0xc0000000 0x00001000>;

    }; //end unknown@0x000000000 (test_subsys_pcie)

    I expect the of_pci_range_parser_init() function to figure out the ranges correctly in the DTS. Is there anything wrong with the way I've declared the "ranges" in DTS ? If the parsing by of_pci_range_parser_init() is wrong, it should've returned an error, but it didn't; It succeeded; But the loop for_each_of_pci_range(&parser, &range) fails to take off !

    Inside of_pci_range_parser_init(), I printed the the foll.:

    parser->range=c1ff789c, parser->end=c1ff78a8, len=12

    The difference between 0x c1ff789c and 0x c1ff78a8 is 12 i.e. 12 bytes. But what does this length indicate??

    The parser structure is defined in linux/of_address.h as follows:

    struct of_pci_range_parser {

    struct device_node *node;

    const __be32 *range;

    const __be32 *end;

    int np;

    int pna;

    };

    What does the range and end mean here? I want to understand this w.r.to the ranges I've declared

    ranges = <0x00000000 0xc0000000 0x00001000>.

    In the loop for_each_of_pci_range(&parser, &range), the 2nd argument is a pointer to the foll. Structure:

    struct of_pci_range {

    u32 pci_space;

    u64 pci_addr;

    u64 cpu_addr;

    u64 size;

    u32 flags;

    };

    Similary I want to understand the of_pci_range structure w.r.to the "ranges" property in DTS.

    What is the link among struct of_pci_range_parser, struct of_pci_range, and "ranges" property in DTS? Going through the code is pretty confusing. Can someone explain?